Automattic is hiring a Marketing Data Analyst

We’re now accepting applications for a new Marketing Data Analyst position at Automattic that might interest some of you:

https://automattic.com/work-with-us/marketing-data-analyst/

In this role you’d be helping us use data and analytics to guide the direction of our marketing efforts on WordPress.com.

Here’s the official description:

Automattic is looking for a data analyst to join the marketing team. You will distill data into actionable insights to guide our customer marketing and retention strategy as well as inform product development efforts.

Primary responsibilities include:

  • Build and maintain standardized reporting on key metrics across the customer lifecycle.
  • Develop customer segmentation models to inform tailored, multi-channel marketing strategies.
  • Conduct ad hoc analyses to better understand customer behavior, needs, and individual test results.
  • Partner with other analysts and developers to increase data accessibility across the organization.
  • Design a process for prioritizing and communicating data requests and priorities.

You:

  • Are proficient in SQL and Excel.
  • Have experience with web analytics platforms such as: Google Analytics, KISSmetrics, or Mixpanel.
  • Have experience working with marketing teams to deliver analyses and answer business questions.
  • Are able to communicate data in a way that is easy to understand and presents clear recommendations.
  • Are highly collaborative and experienced in working with business owners, executives, developers and creatives to discuss data, strategy and tests.
  • Have excellent prioritization and communication skills.
  • Ideally, have web development experience (though not required).

Like all positions at Automattic, you’ll work remotely, and can be based wherever you live. It’s not a requirement that you live in or relocate to the Bay Area to do this job.

If this sounds interesting to you (and how could it not?!?) there are instructions at the bottom of the job description about how to apply.

And if you have any questions about Automattic or this data analyst position, feel free to drop me an email: mhmazur@automattic.com.

 

Visualizing Your SaaS Plan Cancellation Curves

If you work on a SaaS product, you probably have a good idea of what its cancellation rates are, but chances are you don’t know how that changes over time. For example, what % of users cancel after 1 day? How about after 7 days, 30 days, etc?

I worked on a project at Automattic this week to help us understand the cancellation curves for WordPress.com’s plans and am open sourcing the R script so anyone can do the same for their service.

Here’s an example of how the cancellation curves might look for a service with a Gold, Silver, and Bronze plan:

example.png

We can see that most users who cancel do so pretty quickly and that long term about 30% of Gold plan, 20% of Silver plan, and 10% of Bronze plan subscriptions wind up cancelled.

To generate this data for your own product, you’ll just need three data points for each subscription: when it was purchased, when it was cancelled (if it was), and the name of the subscription. The script will take care of analyzing the data and generating the visualization.

You can check out the script and additional details on GitHub here: Cancellation Curve Visualizer.

If you have any questions or run into any issues, don’t hesitate to drop me a note.

The impact of GoDaddy’s 5-year default domain registration option

Andrew Allemann has a great post on Domain Name Wire where he tries to estimate the impact of GoDaddy’s five-year default domain name registration option.

GoDaddy’s shopping cart defaults to a five-year registration period when you place a domain name in your cart. Most people switch this back to just one year, but some don’t. Whether they merely overlook this or decide it makes sense to register the domain name for five years, about 3.5% of new .com registrations at GoDaddy each month are for five years.

Here’s a summary of the math:

  • In June, the .com registry reflects 26,750 five-year registrations which account for 3.48% of all their .com registrations so 26,750/.0348 = 769K total.
  • On average across all registries, only 1.66% of of new .com registrations are for 5 years.
  • Had GoDaddy met the average, it would have only registered 769K * 1.66% = ~12,750 five year registrations.
  • That works out to be a difference of 26,750 – 12,750 = 14K five year registrations or 14K * 5 = 70K years of registrations.
  • Assuming those users would have purchased 1 year registrations if 5 wasn’t the default, that works out to be 70K – 14K = 56K extra years of registrations per month thanks to that five-year default.

Regardless of how you feel about GoDaddy, you’ve got to admit that they’re really effective at upselling.

Hattip to my coworker Wendy for sharing the post.

The Meta Funnel: From User Activity to Product Changes

When we think about funnels, we tend to think about how users move through our product: what percentage of people who visit our homepage sign up, what percentage of those users pay, etc.

If we zoom out, there’s another even more important funnel that we can use to measure how sophisticated an organization is with its data:

User Activity → Data → Analysis → Insights → Product Changes

User Activity → Data: When users interact with your product, are you capturing the relevant data related to their activity? For example, you might track how many people visit your homepage, but how about what percentage scroll below the fold, where they click on the page that they shouldn’t be, the bounce rate, how they’re getting to your site and how that’s changing over time, etc. What’s important to track will vary by product and not everything you track will be important, but if you’re not recording the data, it will be impossible to analyze it.

Data → Analysis: You have data, but is anyone looking at that data regularly? All of the data in the world doesn’t matter if no one ever analyzes it. For some types of analysis your analytics tools will make this step easy, but it also might require complex queries or scripts depending on what questions you’re trying to answer.

Analysis → Insights: One of the hardest things about analytics is that it’s often difficult to look at all of the numbers and draw actionable insights from them. You may discover that your conversion rate is 15%, but is that good or bad? If it goes up to 20% is that because your product has improved or just because the quality of your traffic has changed. If the top term your users are searching for on your support page is “domains”, is that an indication that you need to improve the instructions you provide users in your product, or just an inevitable result of domains being very complex?

Insights → Product Changes: And finally, once you have insights from the analysis you’ve done, are you making any changes to your product as a result? Maybe insights into your product’s support search terms indicate that you do need to improve the guidance you provide to users within your product. Does your team then execute on that by actually improving the guidance within your product?

In my experience with both many years of side projects and at work, the conversion rate across this entire funnel is typically very low. Part of it is just the nature of the beast: it’s hard to set up tracking to collect everything that’s important, it’s hard to analyze the data you do collect, it’s hard to come up with insights from that analysis, and it’s hard to make changes to your product when you do have those insights.

But just because it’s hard doesn’t mean it’s not worth optimizing. If you can double your organization’s conversion rate between any of these steps, it should double the number of improvements you wind up making to your product as a result.

One thing that can help is to discuss with your team and document your organization’s processes for each of these steps. Things like:

  • Who is responsible for implementing analytics on your team?
  • If they don’t have experience setting it up, where can they go to learn?
  • Where can they go to learn what data is important to collect?
  • How do they analyze the data?
  • How do you ensure people are looking at the data often enough?
  • Can you automate the reporting? Should you?
  • Who on the team needs to be involved to maximize the number of insights you’re discovering from your analysis?
  • What does your process look like for turning those insights into actual product changes?

There’s probably a lot of low hanging fruit here for your team to work on. The better your team gets at moving down this funnel, the more improvements you’ll make to your product leading to happier users and more impact on your company’s bottom line.

Luca on Data and Marketing

My friend and coworker Luca Sartoni participated in an AMA on ManageWP where he was asked about using data to guide marketing efforts.

His response explains well why making data-informed decisions will lead to better results than making data-driven decisions:

Interconnecting data science with marketing is one of the keystones of what is commonly called growth hacking. I’m a not a fan of growth hacking by itself though, because business is more complex than just hacking things.

When your decisions are solely based of data science you can call yourself data-driven, but there are tremendous blind-spots. Data looks neutral, but in reality it’s not. The way you plan your data sources, you collect the data, and you process such data to get answers is not neutral and impacts on the final outcome, therefore the information you extract cannot be considered neutral. If you drive your decision-making process solely with that information, your think you are considering data, but in reality you are considering the whole not-neutral process.

For this reason you need to keep in mind more elements when it’s time to make a decision, where data is one of them. There is vision, mission, experience, data, and other elements. When you mix them, giving data the right amount of value, you make a data-informed decision, which is usually better than data-driven, simply because it’s more informed by the other elements.

The whole AMA is worth checking out.

Understand your metrics better by trying to predict how changes will impact them

A few weeks ago, right before launching a new email marketing campaign at work, I asked folks on our team for any predictions about what the email’s conversion rate would be. It wasn’t intended as a serious exercise – more in line with the game where you guess the date a baby will be born or how many marbles are in a jar.

Someone threw out a guess and then a few more people followed with their own. We launched the email campaign and waited. When the results came in, the actual conversion rate was much lower than any of us had expected. We were all surprised and I think a big part of that was due to the guesses we had made beforehand.

I’ve been thinking a lot about that experience and what we can learn from it.

Trying to predict how your metrics will change is a great way to force you to think deeply about those metrics. It’s easy after you make a change to your product to look at a metric and conclude that the change – or lack thereof – makes sense. But if you have to make a prediction beforehand, you’re forced to consider all of the factors. In this example, what do our email conversion rates typically look like? Is it 0.5% of 2% of 5%? Have we tried something similar in the past? Are there other channels that we can learn from? Do we expect this new campaign to have higher or lower conversion rates?

The more you try to predict how your metrics will change, the better you’ll get at predicting them and the better you get, the more likely it is you’ll be able to move those metrics. Imagine two teams, one that always tries to predict what the metrics will look like and one that doesn’t. After doing this several times, which team do you think will be better at understanding how to influence those metrics?

Don’t ask people to guess in public because the subsequent guesses will tend to be anchored towards the first guess. If no one has an idea initially and someone guesses that a metric will be 5%, then people will tend to anchor their guesses around 5%: 4%, 6%, etc. It’s much less likely someone will say 0.4% even if they would have guessed that without seeing any other responses. The easiest way to get around this is probably sending out a survey and have people guess there. Afterwards, you can reveal who was closest.

Put your guesses in writing. It may take some time for the results to come in and it’s easy to forget what your original prediction was. Also, it will make you take the exercise more seriously because of the extra accountability that putting it in writing adds.

Try to predict how your metrics will change before embarking on a project. For example, before we set up this email campaign, should we have tried to predict how it would perform? I think so. Not only would it cause us to think more carefully about the objectives and how to achieve them, but it may have led us not to pursue it in the first place.

Look for other opportunies to make predictions. This doesn’t have to be limited to product changes. For example, lets say you’re about to publish a big announcement on your blog:

  • How many visitors will that post bring over the first 24 hours? How about a week later?
  • How many people will share it on social media?
  • How many of those people will convert into free/paying customers?

A lot of what we do online can be quantified – keep and eye out for ways to hone your predictive skills.

Do any of you all do this type of thing on a regular basis? I’d love to hear about your experience and what you’ve learned from it.

Precision and metrics

I spend more time than I’d like to admit investigating why metrics that should be identical vary from one tracking source to another. Sometimes the difference is due to bugs, but often just due to the nature of how the tracking system works. It’s also given me an appreciation for just how difficult it is to get truly precise metrics.

For example, consider this simple example:

You run a site that where people sign up for an account and then create a post. What percentage of accounts create a post?

Sounds pretty simple, right? Here are two possible approaches and where they can lead your metrics astray:

Using your database to figure out the answer

With this approach, we look at how many accounts there are, figure out how many of them have a post, and calculate the percentage that way. If there were 1,000 accounts created in August and 200 of them have a post, 20% have created a post (yay math).

But… what about users who create a post then delete the post? It’s possible that 20 people wound up creating a post then deleting it, so the “true” number of accounts that created a post is actually 220, but your database only reflects 200 posts so you’ll wind up reporting 20% instead of 22%.

What about users who deleted their account? Instead of 1,000 accounts there were actually 1,100 accounts, but 100 of their owners wound up deleting their account. Unless you set up some special tracking for this, you’re now in a spot where you don’t know exactly how many accounts were created or how many of them created posts.

If your account records have a unique id that increments by 1 each time a new one is created you could figure out the number of accounts relying on that, maybe.

One way around this is to not let people delete their posts or accounts. Maybe you tell them their post was deleted, but don’t actually delete it from your database. Just set a “deleted” flag on the database record and don’t show it to the user. Same thing for accounts. But what happens if they think they deleted their account then try to sign up again with the same email? Will you tell them that their account already exists even though supposedly you deleted it? There are technical solutions around this, but things are getting pretty complicated already.

Using an analytics tool to figure out the answer

Another approach is to use a tool like Mixpanel or KISSmetrics to try to answer the question. Set up a funnel to measure the conversion rate from your Sign Up event to the Publish Post event.

The good news here is that if users delete a post, it won’t impact the conversion rate that the funnel reports.

The bad news is that if they create additional accounts, they’ll only count once in the funnel because funnels try to measure the actions of people, not accounts.

One possible solution

If I really, really wanted to answer this question precisely, I’d set up a new database table that keeps track of accounts and whether they’ve created a post. These records wouldn’t be impacted if the post or account gets deleted so we can trust that the data is accurate.

Alternatively, you can simply redefine your metric so that you can be precise: instead of “what % of accounts created a post?” you ask “what % of non-deleted accounts have a non-deleted post?”. Not very elegant, but far easier to answer than the original metric.

Does perfect precision matter?

I’d argue that for most metrics (with the exception of revenue metrics), being perfectly precise is not critical. The metrics are probably fine as long as they’re close to the true value and that the way you calculate it is consistent over time.

tl;dr: data analysis is fun :).