Friday Updates: Moving Fast and Breaking Preceden, Interviewing, Story of Your Life, the Last Cruise

What I’m working on at Preceden

Upgrading Postgres

I received an automated email from Heroku informing me that Preceden was running Postgres 9.6 which would be deprecated later this year. I read through the documentation detailing the upgrade steps which mentioned there would be some downtime but optimistically imagined it would be in the order of ~10 minutes, but it would up being 45 minutes. I performed the upgrade on a Tuesday morning around 10am EST which meant it impacted a lot of users.

Improving zooming

Preceden lets you zoom into sections of your timeline, but until recently you couldn’t share a link to that zoomed-in section of your timeline. Instead, viewers would always see the full zoomed-out timeline. Similarly, if you were viewing a zoomed-in portion of the timeline in your browser but then exported it to PDF or PNG, those exports were entirely zoomed out.

I worked on a project to improve this behavior, but rolled it out too quickly and wound up introducing a bunch of bugs. Fortunately Michael (a contractor who helps me with support and QA) wound up catching a lot of them.

Optimizing performance

Every few years I take a few days and try to optimize Preceden’s performance. For small timelines, Preceden and nice and speedy but for larger ones (with like 50+ events) it can take 5-10 seconds or more to render the timeline. Every time you make a change to the timeline it has to re-render so this can lead to a lot of wait time for users with large timelines.

I had never used Chrome’s Performance tools before, but watched some videos and tested it out on Preceden. The flamegraph showing the execution time of JavaScript functions wound up being super useful for identifying why large timelines were taking so long to render.

One of the big issues is that behind the scenes Preceden has to figure out how many pixels wide each event name is when it’s displayed on the timeline. The same text with the same font name and size can have different widths on different computers, so it’s not something I can just save with the event and re-use. It turns out though, measuring the width (by creating a hidden DOM element with the text and grabbing its width) takes some time, and when your timeline has hundreds of events it can really add up.

My solution was to save the name widths in Local Storage after they’ve been calculated that way Preceden can try grabbing the saved width from there before creating the hidden DOM element. The text is hashed to minimize storage and also for privacy reasons:

I wound up optimizing a ton of things like this. During the course of fixing this I realized that some previously-built front-end caching functionality had been broken for years. A few days later I realized that Preceden’s backend caching had also been broken for years. So basically Preceden has had zero caching for a long time and I never realized it.

Needless to say, fixing the caching and optimizing the front-end performance has had a big impact on performance. Timelines that previously took 10+ seconds to load are often now taking less than a second which translates to much less wait time for users.

But once again, I didn’t test these changes throughly enough before pushing them to production. Not only that, I pushed them and then took a break for lunch without verifying everything went smoothly. I returned to an overflowing support inbox stemming from various issues:

In the end there wound up being 4 separate bugs. Two I wound have caught with more testing locally, and two I wound have found if I had tested the changes on my staging server.

Moving fast and breaking things

A recurring theme in these mishaps is an an unfounded sense of urgency I have around shipping things. I lean heavily towards the move fast and break things style of entrepreneurship, but lately I think I’m trying to move too fast and it’s bordering on sloppy.

I should have given notice and performed the Postgres upgrade early on a Sunday morning to minimize the impact.
I shouldn’t have launched an incomplete iteration on the zooming features
I should have testing the caching changes on the staging server before shipping to production

Some extra testing and QA could go a long way without slowing things down much.

What I’m working on at Help Scout

Last week I led the first round of interviews to hire a new Data Lead. A coworker (KJ) and I interviewed 7 people, several of whom have moved onto additional interviews and a trial project. In the coming weeks we should wind up hiring one of them to take ownership of the data function at Help Scout.

I don’t have a ton of experience interviewing, but thankfully Help Scout provided some helpful interview training beforehand. One thing they didn’t talk about but which I learned over the course of the week is that interviewing is exhausting, at least to me. Probably part of why I lean towards individual contributor/solo entrepreneur roles.

What I’m reading

I finished Ted Chiang’s collection of stories released as Exhalation. I really enjoyed these stories and have moved onto Story of Your Life and Others, another collection of short stories by him.