A/B Testing Preceden’s Homepage – Round 1: 37% Improvement

Preceden’s homepage can currently be divided into four sections (see below):

1) The splash screen (Make an amazing timeline with Preceden, Quote, Sign Up)

2) Example

3) Features

4) Featured On

I’ve been running an A/B test to determine the impact of the splash screen design and the features.

Here are the results along with their corresponding conversions rates. The conversion rates reflect the % of people that make it to the sign up page. (In future tests I’m going to measure the % that actually sign up, which is probably a better indication of a successful conversion.)

Note that I’m stopping this A/B test before Google Website Optimizer has reported a statistically significant result, but I’m eager to try additional variations and don’t want to wait any longer. So take these results with a grain of salt.

Original

Conversion rate: 17.2%

Combination 1: New splash screen

Conversion rate: 19.2% (11.4% improvement)

Combination 2: Original splash screen, no features

Conversion rate: 23.7% (37.3% improvement)

Combination 3: New splash screen, no features

Conversion rate: 18.3% (5.95% improvement)

Summary

Page Sections

###

The results were 180 degrees from what I expected. I’d have bet money that:

a) The sexier splash screen and call to action had a higher conversion rate. In fact, it decreased the conversion rate by 8.3%.

b) The shiny list of features would have had a higher conversion rate. In fact, not including the features increased the conversion rate by 15.1%.

The best result was with the original splash screen and without the list of features, which yielded a 37% higher conversion rate. Why? My theory is that in both cases, the landing page was simpler and people generally prefer simpler over more complex. Makes you appreciate why everything Google designs is dead-simple.

More to follow…

AdWords: Week 1 Results

Last Sunday I started advertising Preceden via Google AdWords. I’ve played around with it before, but never seriously.

I wrote a few ads, picked a bunch of timeline-maker keywords to target, set my daily budget to $3 (and a day or two later $10) and flipped the switch.

Week 1 Results

Impressions: 18,439

Clicks: 564 (3.06%)

Visitors: 548

CPC: $0.13

Cost = $72.16

Sign Ups: 3

Revenue: $87

Profit: $14.84 (21% ROI)

Thoughts

The variance of this whole process is extremely high. 3/548 is a 0.54% conversion rate. Good, bad… I don’t know. It seems bad, but I don’t really have a lot to compare it to.

The good news is that the campaign made money. Not much, but something. Had 2 people upgraded instead of 3, it would have lost money (-20% ROI). Had one additional person signed up, the return would be exceptional (61% ROI). This whole thing makes you appreciate how small improvements to your site, which in turn might convince a few more people to upgrade, can make a huge difference to your bottom line.

Next steps: Refine A/B tests, bump daily budget to $30, and see what happens.

Here are the keyword details:

How to Calculate Your AdWords ROI for a Freemium, One Time Purchase Web App

I run a web-based timeline tool called Preceden and am about to start experimenting with AdWords to bring in traffic. What kind of ROI can I expect given my current conversion rates and price?

Let:

s = Conversion rate representing how many new visitors sign up for a free account

u = Conversion rate representing how many free account holders upgrade to a paid account

p = The price of the web app

c = The average Cost Per Click (CPC) for our AdWords campaign

For example, let’s say our CPC is $0.25, 25% of the new visitors sign up for an account, 4% upgrade to a paid account, and upgrading costs $29 (which is roughly my situation with Preceden, give or take).

First, we’ll calculate the Expected Value for a single user. If 25% of the visitors sign up for a free account, and 4% upgrade to a paid account, the overall conversion rate is 1%. So 1% of the time a visitor signs up and upgrades to a paid account and we make $29 minus the cost of AdWords ($0.25). The other 99% of the time the user doesn’t sign up or doesn’t upgrade their account and we lose the $0.25.  The Expected Value then is 1% * $28.75 + 99% * -$0.25 = $0.04. In other words, we can expect to make about 4 cents per AdWords visitor.

Our ROI is the Expected Value ($0.04) divided by the cost ($0.25) or 16%.

If these numbers are correct, then if we spend $100 on AdWords, we can expect to make $116. If we spend $1,000 we can expect to make $1160. And so on.

The last line simplifies it all a bit. The implications make intuitive sense: if we increase the conversion rates or the price, we can expect to make more money. If we decrease the CPC, we also make more money. Likewise, if we want to double our ROI we can either double the price ($29 to $58), half the CPC ($0.25 to $0.125), or double the conversion rate (1% to 2%). Which of the three do you think is the easiest to do?

###

Over the next few months I’m going to work on measuring and optimizing Preceden’s ROI with eye towards maximizing its profits. I’ll provide regular updates — stay tuned.

February in Review: Life Tracking, Major jMockups Updates, and Preceden’s Best Month Yet

In December I announced a $4K/month challenge, where I try to reach $4K/month in profit from my web apps by the end of 2011.

In this post I’ll give an overview of what happened in February. Click here to see January’s report.

By the numbers:

1) According to RescueTime, I spent 121 hours working over the course of the month, which equals about 30 hours/week or 4.3 hours/day. (Not including day job.)

2) 10,578 people visited Preceden. 529 people signed up for an account. 20 people upgraded to a paid account. Revenue: $580. Total time spent working on Preceden: 0 hours.

3) 5,303 people visited jMockups. 302 people signed up for an account. None upgraded. Revenue: $19.

4) 9 blog posts:

Three on this blog:

jMockups and Preceden January Review – 5 Feb

Seeking Alpha Testers for jMockups Import Tool (cross post from the jMockups Blog) – 6 Feb

Launch! jMockups Website to Mockup Converter – 19 Feb

Five on the jMockups Blog:

New Keyboard Shortcuts to Make Designing Mockups Easier – 3 Feb

Seeking Alpha Testers for new jMockups Import Tool – 6 Feb

New Homepage Design, Some A/B Testing, and Why Data Can Be Misleading – 11 Feb

Website to Mockup Converter Launched – 19 Feb

Reengineering jMockups for Optimal Performance – 26 Feb

And one as a guest post on Sebastian Marshall’s blog:

Nine Tips for Getting Started with Life Tracking

Overview:

In terms of progress, February was a huge month for jMockups. For a good chunk of December, January, and early February, I was busily building and preparing for the launch of the Website to Mockup converter. The official launch was on Saturday, 19 Feb while I was in NYC with Chris Conley and Mike Nicholaides. All in all, it went fairly smoothly, though there were some issues rooted in the slowness of the jMockups editor. The week following the launch I reengineered the editor to eliminate the performance issues. Today was another huge update: the launch of the redesigned mockup editor. In a nutshell, compared to a month ago jMockups has a sexier design, an improved user interface, vastly improved performance, and one major new feature: the Website to Mockup converter.

Meanwhile, Preceden had its best month yet. 20 paid signups resulting in almost $600 in revenue. Long term I believe that jMockups has much more potential than Preceden, yet it’s a little unnerving that the thing I’m spending so much time on is hardly making any money and the thing I’ve ignored for the last seven months is growing and bringing in decent revenue. Rather than spend 100% of my time on jMockups and none on Preceden, I’m going to aim for a 85/15 ratio in the future. I keep saying it, but a few A/B tests, SEO, and target AdWords campaigns could have a huge affect on Preceden’s revenue so it’s time to start doing something about it.

Finally, I’ve been doing life tracking for 2+ months now and have had some great results from it. Tracking + slow, incremental improvements go a long, long way. Check out my post on Sebastian’s blog for more details.

One last thing: I really enjoy talking with folks about their startups and poker bot work, so if there’s anything I can help you with or you just want to say hey, please shoot me an email.

Thanks for reading — Matt

jMockups and Preceden January Review

As promised in my 2010, 2011, and a $4K/month challenge post, I’m going to do a quick recap each month of the progress I’ve made with towards achieving that end.

Here’s what happened in January.

Preceden

Total Sign Ups: 510

Upgrades to Preceden Pro: 18 (3.5%)

Revenue: $522

Progress: I spent a total of about 5 hours working on Preceden in January, which was split between reverting December’s pricing plan changes and with integrating Mixpanel last week to start getting better insights into how people are using the tool. To date I haven’t spent any time trying to optimize the conversion funnel, which gives me high hopes that I can at least double it with some calculating A/B tests.

jMockups

Total Sign Ups: 135

Upgrades to jMockups Pro: 0

Revenue: $19

Progress: I spent a lot of time working on jMockups in January, with a focus on improving the user experience, fixing bugs, and making progress towards launching the new website import tool later this month.

Most of the major changes were covered on the jMockups Blog:

In early January I began working with a talented San Fransisco-based JavaScript developer on jMockups with the intent of eventually bringing him on as a co-founder if the partnership went well. We got along extremely well and I was thrilled with how quickly he was picking up the code, but personal circumstances prevented him from continuing. So back to one.

jMockups is somewhere in the trough of sorrow right now:

And there’s even been several crashes of ineptitude thanks to some poor QA practices…. so I’m on my way :)

My primary goal in February is to launch the new website import tool, which will let you import any existing website into jMockups, allowing you to redesign and share it in seconds.

Before I launch it (and it’s 95% ready), I want to fix a few bugs, improve a few existing features, and set up some more in depth analytics to prepare for the potential influx of visitors after I launch it. There’s nothing worse than launching an amazing new feature only to have 50% of the new visitors run into a bug that sends them away with a poor taste in their mouth…. except for never launching it at all, I suppose.

According to RescueTime, I spent 120 hours working in January, or about 27 hours/week.

Life

January was kind unique month for me because of how much time I spent on the road for my day job: 11 days completely away from home and another 6 partially away. 27 hours/week is a bit high for me — average is about 20 to 25. The extra hours were only possible because I was traveling and did not have anything else to split my time with.

On New Years Day I began doing daily life tracking, which was inspired by several posts Sebastian Marshall has written on the topic. I’ve been tracking how much I eat out, how much I exercise, how much I spend, my finances, sleep schedule, etc. This deserves its own post, so I’ll save the details for later, but in a nutshell it’s given me some astounding insights into my life (positive and negative) and by making a few minor changes I’ve already had incredible gains from it. I highly encourage you to try it out if you’re looking to improve some aspects of your life (start with Sebastian’s posts).

With that, back to coding.

I Eliminated the Free Plan from my Web App for a Month: Here’s What Happened.

Background

In January 2010 I launched Preceden, a free web-based timeline tool. In May I migrated to a freemium plan, where users could add a certain number of events for free but had to upgrade to the pro plan in order to create more than that. After some experimentation with different limits and prices, I settled on 5 events for free and a one time payment of $30 to upgrade to the pro plan.

In June I started working on jMockups, a web-based high fidelity mockup tool. From June until the beginning of December I hardly touched Preceden. At the beginning of December I started reviewing Preceden’s stats and realized Hey, this is actually bringing in some decent money so I decided to start dedicating some time each week to working on it. (For a more detailed explanation of why I wasn’t watching it closely, read my 2010, 2011, and a $4K/month challenge post.)

Removing the Free Plan

Looking over the revenue figures I wondered: What would happen if I completely removed the free plan? Instead of a user being able to create 5 events for free, new users would have to pay first to use it.

After about three minutes of planning, I decided to go ahead and implement the change.

Additional Changes

I wanted to change as little as possible so that the results of the experiment would be accurate. After all, you can’t change several variables at once and then attribute the changes to only one.

That being said, I made a few small changes to the site which may have affected the results.

1) I dropped the price from $30 to $29

2) I changed the background color the header from #112a40 (dark blue) to #244386 (a lighter blue) and reduced its height

3) I made some layout and copy changes on the homepage

4) I removed the reCAPTCHA form from the sign up page

I shouldn’t have made these changes, but at the time I was more focused on the potential increase in revenue than on running a proper experiment.

Before and After Screenshots

Click to view full size:

Homepage

Sign Up Page

Notice at the top of the new sign up page it indicates what step of the process you are on (1. Create an Account, 2. Checkout, 3. Create Timelines). I also changed the copy on the button from “Create my Account” to “Create Account and Proceed to Checkout”. The wording will come into play later.

Checkout Page

(Only present on the pay-first version)

Results and Analysis

Notes

  • The number of homepage and sign up page visitors was taken from Google Analytics by looking a the number of new visitors to those particular pages.
  • Promotions refers to users I upgraded because they are bloggers and bloggers can get upgraded to the pro account for free.

Analysis

From Nov 1 – Dec 4, 3287 new visitors came to the Preceden homepage, 894 visited the sign up page, 567 created an account, and 22 upgraded to a paid account. That’s a 0.67% conversion rate from new visitor to paid user.

From Dec 6 – Jan 1, 3343 new visitors came to the site, 528 visited the sign up page, 210 created an account (oops), and 5 upgraded to a paid account. That’s a 0.15% conversion rate.

Users who were able to test Preceden before paying were 347% more likely to upgrade to a paid account than those who weren’t.

Also — and this is definitely my fault — 205 people filled out the sign up page and then abandoned the process on the Checkout page. Why? Most people probably didn’t realize they had to pay before using it. Sure, the Pricing page said “$29 for Unlimted Access” without mentioning a free account and Step #2 of the sign up process was called “Checkout”, but overall the copy didn’t clearly indicate that you had to pay before you could use the app.

Another problem is that the site doesn’t include a demo. It includes examples, but nothing the user can create on their own. Adding a demo section so that new users can get that “Oh cool” moment would have gone a long way.

The good news is that about 4% of the people who created an account converted to the paid plan when they were able to test the product first. Even if the pay-first experiment resulted in better figures, I feel more comfortable letting users test the product before paying. I think this depends a lot on your app and your business model, but for a tool like Preceden there’s no reason not to let people try it for free first. Accordingly, I’m switching back to the freemium version starting today.

Lessons Learned

  • If you run a web app and start building a new product, don’t neglect what you’ve already built. Dedicate some time each week to analyzing your key performance indicators and to making small changes — they’ll add up in the long run.
  • The best way to test this would have been to run the experiment on a fraction of visitors to the site. For example, show a pay-only version to half of the visitors and a free plan version to the other half.
  • Don’t change several variables at once when you run an experiment. Every change you make will affect the results.
  • The time of the year matters. Measuring only the number of paid users would be a bad way choice because a lot of the customers are teachers and I don’t imagine a lot are on the market for a new teaching tool in December. Measuring the conversion rates is better, but there still may be a “Oh I’ll wait till after the holidays to pursue this” mentality among some of the visitors, which could have affected the results. Running this experiment simultaneously would have yielded better results.

Over the next few weeks I’m going to experiment with A/B testing and AdWords to see what’s possible. I’m also going to email the 205 people who signed up during this experiment but didn’t pay; I’m sure they’d be interested in trying it out.

Can you think of anything else that might improve the conversion rates? Leave a comment below and I may test it out.

###

If you enjoyed this post, you should subscribe to the RSS feed for this blog to get future updates. Also, if you’re a web designer or developer I’d love to get your thoughts on jMockups, my new web design startup. Drop me a note at matthew.h.mazur@gmail.com or @jmockups on Twitter. Thanks!

2010, 2011 & a $4K/month Challenge

A few months back with what seemed like two months of work left to do, I publicly committed myself to launching jMockups in four weeks. The number of things I wanted to get done prior to launching was daunting, but I desperately needed to get the product out there and in the hands of actual users to start collecting feedback.

How’d it go? I didn’t launch it in four weeks; I launched in less than three.

Public accountability is a powerful thing. Sebastian Marshal writes about his own experience:

If I hadn’t set this goal and been accountable publicly, to my friend and to everyone who reads here, I wouldn’t have done it in two weeks. Honestly – I’m pretty internally motivated, but I’ve had a lot of stuff going on the last two weeks, it wouldn’t have happened. But it did happen, largely because I was publicly accountable.

This productivity hack that works especially well for me. I’m an INTJ on the Meyers Briggs type indicator and one of our defining characteristics is the value we place on competence. By publicly committing myself to launch, I put my own competence on the line so I did what it took to meet the goal. Practically this meant postponing a lot of the things on my todo list until after the launch, working some long nights and weekends, and taking a week off of work.

Publicly committing myself worked well for me once, so I’m trying it again.

The Scene

I spent most of 2010 building. I launched Preceden in late January and continued working on it through May. In June I started working on jMockups and launched it in late October. My focus in 2010 (and largely 2009 and 2008) was on learning and building. I knew I eventually wanted to make money off of the apps, but money wasn’t my primary goal.

I’ll be with my current job until October 2012 so I can’t pursue this full time just yet. Because I have a day job I’ve never had to rely on my web app income to survive. It’s always been like Oh, I had another sign up. That’s nice. That’s got to change.

October 2010–a mere 21 months away–will approach fast. In order for web app development/entrepreneurship to be viable long term, I have to start making more money. My true passion lies in building things–not money–but without money I can’t spend my time building things.

The Goal

I don’t have exact numbers (which is part of the problem), but Preceden currently makes about $500/month and costs $70 to operate. jMockups makes $24/month (whohoo!) and costs about $200/month to operate (doh!). Taken together, I’m making about $250/month.

It’s not entirely fair to value the apps in terms of their current profits, but that’s obviously a big part of it. Preceden targets a small niche and has a small goal: be the best timeline tool. jMockups targets a large niche and has an ambitious goal: improve the way people design websites. Long term, jMockups has the potential to be a home run; Preceden doesn’t.

With that in mind:

My goal is to make $4,000 per month from Preceden and jMockups by the end of 2011.

That’s about 16x what they make now. If a public commitment isn’t scary, it’s probably not ambitious enough. And since this is terrifying, I figure it’s a good number to shoot for.

I’ll make monthly progress updates starting at the end of January.

The Plan

Preceden has a marketing problem. It’s is a quality tool that has a lot of happy users, but not enough people know about it. I need to get more people to the site and need more of them to convert to paying customers. My plan with Preceden is to start marketing it heavily (via things like AdWords), perform lots of A/B testing, and optimize the hell out of it by way of extensive analysis. If I can get Preceden to a point where outbound marketing has a measurable positive ROI, I’ll be in really good shape.

jMockups has a product problem. The tool is good, but not great. Trying to change the way people design websites is hard (I probably should have picked a more narrow niche to start with, but that’s another story). I’ve been adding two or three new features a week since it launched in October, but I haven’t spent much time on the other things it takes to create a successful web product. For example, there’s currently way too much friction from when a user arrives at the site and to when they create a mockup that they’re happy with. And it shows in the usage metrics (75% of new users create 1 mockup and never come back). In 2011, I’ll continue working on the product but I’ll place a stronger emphasis on usability, education, and building a community. The revenue should follow from doing these things well.

I have a sole founder problem. But not really. I like the independence of working alone, but having someone else to build with and bounce ideas off of would be great. I’m not going to spend a lot of time actively searching for a cofounder, but if the opportunity presents itself I’d definitely give it a shot. (Interested? Drop me a note: matthew.h.mazur@gmail.com)

###

So here’s to 2011. I don’t know how things are going to turn out, but hopefully with this public commitment they’ll turn out a little bit better than they otherwise would have.