Background
In January 2010 I launched Preceden, a free web-based timeline tool. In May I migrated to a freemium plan, where users could add a certain number of events for free but had to upgrade to the pro plan in order to create more than that. After some experimentation with different limits and prices, I settled on 5 events for free and a one time payment of $30 to upgrade to the pro plan.
In June I started working on jMockups, a web-based high fidelity mockup tool. From June until the beginning of December I hardly touched Preceden. At the beginning of December I started reviewing Preceden’s stats and realized Hey, this is actually bringing in some decent money so I decided to start dedicating some time each week to working on it. (For a more detailed explanation of why I wasn’t watching it closely, read my 2010, 2011, and a $4K/month challenge post.)
Removing the Free Plan
Looking over the revenue figures I wondered: What would happen if I completely removed the free plan? Instead of a user being able to create 5 events for free, new users would have to pay first to use it.
After about three minutes of planning, I decided to go ahead and implement the change.
Additional Changes
I wanted to change as little as possible so that the results of the experiment would be accurate. After all, you can’t change several variables at once and then attribute the changes to only one.
That being said, I made a few small changes to the site which may have affected the results.
1) I dropped the price from $30 to $29
2) I changed the background color the header from #112a40 (dark blue) to #244386 (a lighter blue) and reduced its height
3) I made some layout and copy changes on the homepage
4) I removed the reCAPTCHA form from the sign up page
I shouldn’t have made these changes, but at the time I was more focused on the potential increase in revenue than on running a proper experiment.
Before and After Screenshots
Click to view full size:
Homepage
Sign Up Page
Notice at the top of the new sign up page it indicates what step of the process you are on (1. Create an Account, 2. Checkout, 3. Create Timelines). I also changed the copy on the button from “Create my Account” to “Create Account and Proceed to Checkout”. The wording will come into play later.
Checkout Page
(Only present on the pay-first version)
Results and Analysis
Notes
- The number of homepage and sign up page visitors was taken from Google Analytics by looking a the number of new visitors to those particular pages.
- Promotions refers to users I upgraded because they are bloggers and bloggers can get upgraded to the pro account for free.
Analysis
From Nov 1 – Dec 4, 3287 new visitors came to the Preceden homepage, 894 visited the sign up page, 567 created an account, and 22 upgraded to a paid account. That’s a 0.67% conversion rate from new visitor to paid user.
From Dec 6 – Jan 1, 3343 new visitors came to the site, 528 visited the sign up page, 210 created an account (oops), and 5 upgraded to a paid account. That’s a 0.15% conversion rate.
Users who were able to test Preceden before paying were 347% more likely to upgrade to a paid account than those who weren’t.
Also — and this is definitely my fault — 205 people filled out the sign up page and then abandoned the process on the Checkout page. Why? Most people probably didn’t realize they had to pay before using it. Sure, the Pricing page said “$29 for Unlimted Access” without mentioning a free account and Step #2 of the sign up process was called “Checkout”, but overall the copy didn’t clearly indicate that you had to pay before you could use the app.
Another problem is that the site doesn’t include a demo. It includes examples, but nothing the user can create on their own. Adding a demo section so that new users can get that “Oh cool” moment would have gone a long way.
The good news is that about 4% of the people who created an account converted to the paid plan when they were able to test the product first. Even if the pay-first experiment resulted in better figures, I feel more comfortable letting users test the product before paying. I think this depends a lot on your app and your business model, but for a tool like Preceden there’s no reason not to let people try it for free first. Accordingly, I’m switching back to the freemium version starting today.
Lessons Learned
- If you run a web app and start building a new product, don’t neglect what you’ve already built. Dedicate some time each week to analyzing your key performance indicators and to making small changes — they’ll add up in the long run.
- The best way to test this would have been to run the experiment on a fraction of visitors to the site. For example, show a pay-only version to half of the visitors and a free plan version to the other half.
- Don’t change several variables at once when you run an experiment. Every change you make will affect the results.
- The time of the year matters. Measuring only the number of paid users would be a bad way choice because a lot of the customers are teachers and I don’t imagine a lot are on the market for a new teaching tool in December. Measuring the conversion rates is better, but there still may be a “Oh I’ll wait till after the holidays to pursue this” mentality among some of the visitors, which could have affected the results. Running this experiment simultaneously would have yielded better results.
Over the next few weeks I’m going to experiment with A/B testing and AdWords to see what’s possible. I’m also going to email the 205 people who signed up during this experiment but didn’t pay; I’m sure they’d be interested in trying it out.
Can you think of anything else that might improve the conversion rates? Leave a comment below and I may test it out.
###
If you enjoyed this post, you should subscribe to the RSS feed for this blog to get future updates. Also, if you’re a web designer or developer I’d love to get your thoughts on jMockups, my new web design startup. Drop me a note at matthew.h.mazur@gmail.com or @jmockups on Twitter. Thanks!
Matt Mazur 
Tht’s a nice lesson learnt. Wish you do great!!
Why don’t you just offer a 30-day free trial? That’s a fairly standard way around this problem:
http://onstartups.com/tabid/3339/bid/33197/The-Dharmesh-Test-16-Questions-For-Better-SaaS-Companies.aspx
###
That’s a great idea — I definitely might try testing it in the coming weeks. Thanks for the suggestion. – Matt
i believe a proper ab test could bring better results.
I agree a live demo with limited access can would help a lot.
And also get testimonials and rating from your old customers and showcase them.
But nothing beats providing a Trial for Free.
testing around xmas is difficult because peoples patterns are disrupted by holidays…. inconclusive test
###
Agreed — there were definitely issues my methodology, the seasonality included. I’ll try to improve in the next test – Matt
This is really interesting, but you need to test results like these for at least 3 months.
I usually “look” at anything I buy for a while before I actually purchase it. Even if there isn’t a free trial, I’ll roll a purchase back-and-forth (especially those that are paid monthly) for weeks. I’ll check out comments, competitors, reviews, user submitted YouTube walkthroughs, etc, before plunking down the cash and committing to something every month.
In my scenario, I would have seen the price change and thought “I should wait to see if any other prices are changing”.
Whatever the reason, testing changes and measuring them for a long period of time is important. Online, time of year, weather, economic mood, trust, etc all play roles in purchase decisions.
37 Signals sheds some light on their pricing decisions in episode 23. For those interested, check it out: http://37signals.com/podcast#episode23
###
Fantastic feedback — thank you so much. I recently picked up a book called Always Be Testing which talks about the different types of customers. I believe you fit into the “Methodical/Analytical” category (same as me). The trick is coming up with a homepage that suites all the different personality types and that’s something I’m going to be experimenting with in the coming weeks.
One thing though — I’m not sure the duration matters. It’s the number of occurrences that is important. For an extreme example, imagine you have a site that gets a million visitors a day. You don’t need to run a test for very long to get good results.
Again, thank you for the feedback. -Matt
Yeah, you gotta have a demo at the very least. I think a trial is best though, so they can actually fully use the product before deciding if it’s worth their money.
I remember reading a post a few months back where someone did something similar (removing their free plan) but they still had a trial or demo or whatever – and the results were quite different. Revenue was up, and considerably so.
Nice to have another perspective on this though. And while you did change some other variables at the same time that certainly had an influence, if anything I would say all of those changes made the sign up process better, not worse. But you can never actually know unless you test them individually. :)
###
I’m giving serious thought to a free trial period — I’ll let you know how it goes.
One mistake I made was not adjusting the copy on the site to reflect the new business model. There’s definitely a different approach you need to take when you nix the free plan.
Gracias – Matt
For your startup, you should try https://www.thesunnytrail.com/ ;-)
###
Very interesting — I’ll check it out. Thanks. – Matt
wow I’m not a web developer or anything but the business lessons learned from your experience and test are great.
##
Thanks! – Matt
Am facing the exact same dilemma for a news aggregator I am developing. Appreciate the insights.
###
Good luck with it. Let me know how it goes. – Matt
Thanks for this generous case study. We delayed providing a freemium sample of our online courses for content authors, but it’s popular. We’ll monitor it much more closely from now on.
Thanks for the insight – I have been really debating this subject with a few friends, but real data makes this a much easier decision. Thanks for sharing Matt!
Hi Matt,
For now, I can’t tell. I decide to go 100% free on http:/www.pocket-locator.com to test donation model. Some says people donate.
For now, I must say it’s a $7 epic fail but since it’s a good demo for my consultant job it’s ok. But I’m looking for a good business idea for such freemium small services.
I have a demo page. 70 members in 3 weeks.
Great post Matt. The lift from freemium is impressive isn’t it!? We shared this on our twitter feed.
Know This When Choosing Freemium or Paid For Your Web App - W3i Blog
» I Eliminated the Free Plan from my Web App for a Month – Tom Altman’s Wedia Conversation
Ye Gods this is Flawed – yet still sooo tasty. Thanks a million – always testing !
Actually I dont really understand why did you try the “pay first” model in the first place. Was it just for the sake of the experiment? For a tool like Preceden, a free web-based timeline tool, which is not know from the manstream, it is difficult for ausers to accept to pay without even knowing how much useful it will be for him. I am even suprised you got registration during you testing, do you think it was people who got recommended by existing users?
Also demo is super important! I would recommend video (<2min) rather than screenshots (quite boring).
Still interesting experiment, thanks for sharing! Cool that you list event the details that you changed in design, these stuff can weirdly influence a lot :-)
Good luck for your 4K usd/m challenge!
It’s time to get paid! « I'm an Undertaker.