Understand your metrics better by trying to predict how changes will impact them

A few weeks ago, right before launching a new email marketing campaign at work, I asked folks on our team for any predictions about what the email’s conversion rate would be. It wasn’t intended as a serious exercise – more in line with the game where you guess the date a baby will be born or how many marbles are in a jar.

Someone threw out a guess and then a few more people followed with their own. We launched the email campaign and waited. When the results came in, the actual conversion rate was much lower than any of us had expected. We were all surprised and I think a big part of that was due to the guesses we had made beforehand.

I’ve been thinking a lot about that experience and what we can learn from it.

Trying to predict how your metrics will change is a great way to force you to think deeply about those metrics. It’s easy after you make a change to your product to look at a metric and conclude that the change – or lack thereof – makes sense. But if you have to make a prediction beforehand, you’re forced to consider all of the factors. In this example, what do our email conversion rates typically look like? Is it 0.5% of 2% of 5%? Have we tried something similar in the past? Are there other channels that we can learn from? Do we expect this new campaign to have higher or lower conversion rates?

The more you try to predict how your metrics will change, the better you’ll get at predicting them and the better you get, the more likely it is you’ll be able to move those metrics. Imagine two teams, one that always tries to predict what the metrics will look like and one that doesn’t. After doing this several times, which team do you think will be better at understanding how to influence those metrics?

Don’t ask people to guess in public because the subsequent guesses will tend to be anchored towards the first guess. If no one has an idea initially and someone guesses that a metric will be 5%, then people will tend to anchor their guesses around 5%: 4%, 6%, etc. It’s much less likely someone will say 0.4% even if they would have guessed that without seeing any other responses. The easiest way to get around this is probably sending out a survey and have people guess there. Afterwards, you can reveal who was closest.

Put your guesses in writing. It may take some time for the results to come in and it’s easy to forget what your original prediction was. Also, it will make you take the exercise more seriously because of the extra accountability that putting it in writing adds.

Try to predict how your metrics will change before embarking on a project. For example, before we set up this email campaign, should we have tried to predict how it would perform? I think so. Not only would it cause us to think more carefully about the objectives and how to achieve them, but it may have led us not to pursue it in the first place.

Look for other opportunies to make predictions. This doesn’t have to be limited to product changes. For example, lets say you’re about to publish a big announcement on your blog:

  • How many visitors will that post bring over the first 24 hours? How about a week later?
  • How many people will share it on social media?
  • How many of those people will convert into free/paying customers?

A lot of what we do online can be quantified – keep and eye out for ways to hone your predictive skills.

Do any of you all do this type of thing on a regular basis? I’d love to hear about your experience and what you’ve learned from it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s