When you ship a major bug before signing off for the day

For better or worse, I still handle all of Preceden’s support requests.

At one point I did have help (thanks Liesl!), but these days support takes at most an hour per week, and all requests fall into two buckets:

  1. Things I can deal with in 30 seconds (like refund requests)
  2. Things that only I can handle, like bug fixes

As a result, there hasn’t been a pressing need to outsource support.

But even though support only takes about an hour per week, I’ve had this bad habit of checking support frequently at night and on weekends. It’s been common for me to pull up the Help Scout app on my phone right before bed, which I think we can all agree is a bad idea.

So, lately, I’ve been trying not to check support after 5pm. It helps me focus on my family after work, and not get sucked into thinking about things right before bed. It’s been a good decision, mostly.

But yesterday around 4pm I rolled out an update to fix a small bug. I tested things on my end, though not thoroughly, because it seemed like a low risk update.

Turns out that my fix caused all timelines using custom fonts (ie, not the default Proxima Nova) to throw an internal server error. And a lot of users use custom fonts.

I didn’t check support before signing off for the day yesterday, and then due to me trying to check support less frequently, I didn’t check it again until this morning, where I was greeted with two dozen support requests about Preceden being broken including:

  • “Hello, I have an urgent request, we are getting into a meeting and see one of our timelines is throwing an error.”
  • “I need access to my timelines! When will they be working again? or when will someone be able to help me?”
  • “Why am I unable to access my timelines? I can see them on the dashboard but when I click into them to access and/or edit I receive an error message!”
  • “Hello, we cant open our timelines. We always get HTTP 500 Error. Could you check this please? We have a meeting today and would like to use our timeline. It would be great if you could fix it today.”

Not great.

I quickly fixed the issue and posted an update, though due to the duration of the issue it was no doubt very frustrating to many of Preceden’s customers.

The question is what to do to avoid it in the future. A few options:

  • Don’t ship in the late afternoon.
  • Check support once in the evening around 8pm to see if anything is on fire.
  • Hire someone to check it for me on nights and weekends, and give them a means of notifying me if there’s a big problem.
  • Hire someone to handle support/documentation, and their responsibilities would include checking it periodically on nights and weekends.

I’d like to avoid hiring someone if I can help it. I can’t think of the last time this happened and I don’t want to make a big change because of an isolated incident, but also do want to avoid it happening again.

Checking once in the evening might be my best option for now. We’ll see.

2 thoughts on “When you ship a major bug before signing off for the day

  1. Another idea: If you have a programmatic way to count incoming support emails (perhaps a HelpScout webhook?), maybe you could set up a threshold alert (2-4x the norm) to text you a heads up when support emails spike? This way you don’t have to pay anyone. If there is an issue you can fix it. If there isn’t a blocking issue, you can just ignore it and will have only wasted 1-2 min of your weekend.

    1. This is a great idea. I’ll see if I can figure out how to do it and if I get it working I’ll write up a separate post explaining how. Thanks for the suggestion!

Leave a comment