Lean Domain Search, the new domain name search engine I recently launched on HackerNews, has received extremely positive reviews, but one complaint that pops up fairly frequently is the color scheme.

Brian Graves recently wrote in the comments of one HackerNews thread:

The green on black is very harsh to my eyes. Have you considered alternate color schemes?

And one other commenter agreed:

I agree with the sentiment of the green on black. It does look harsh and 90’s stylish web 1.0. I would suggest doing A/B testing on some color schemas.

I decided to follow his advice and run an A/B test with different color schemes and see what affect it had, if any, on Lean Domain Search users.

The Battle Between Light and Dark

Before getting into the technical details, I’ll show you what the final versions of the site looked like.

On the left is the original dark version of the homepage, search results page, and about page. On the right is the alternate light version I wanted to test (click to zoom in):

A/Bingo > Google Website Optimizer

I’ve done a bit of A/B testing of headlines and call to action buttons using Google Website Optimizer, but I’ve never A/B tested an entire color scheme across multiple pages. If that’s possible with Google Website Optimizer, I don’t know how to do it. Fortunately, there’s a much better solution for Rails developers: Patrick McKenzie’s A/Bingo.

The two big selling points for me for A/Bingo are:

• You can perform A/B tests in the controller, not just the views
• You can quickly create new tests using only a few lines of code

There are few ways you could A/B test the color scheme using A/Bingo, the simplest being to overwrite your default stylesheet with a new one and measuring the results:

<%= ab_test("test_new_color_scheme", %w{yes no}) do |should_show| %>
   <%= stylesheet_link_tag("new_color_scheme") if should_show == 'yes' %>		
  <% end %>

Because of the way CSS works, any colors that are specified in new_color_scheme.css would take precedence over those in your default stylesheet.

Note that A/Bingo does support a true/false syntax, but using yes/no works better because you can add a test_new_color_scheme=yes parameter to a URL in development so that you can preview the new stylesheet and you can’t currently do that using A/Bingo’s true/false syntax.

bingo! if num_searches == 4

In Lean Domain Search’s case, I wanted to measure the impact on the number of searches a user performed. If the color scheme really was obnoxious, there should be a measurable difference in the number of searches performed. For example, maybe switching to a lighter design encourages people who would otherwise leave because of the harsh design stay longer because they don’t mind the light design as much.

Measuring the average number of searches per user would have been an ideal metric, but with traditional A/B testing you have to identify a single event as your conversion. As such, I chose to measure the percentage of visitors that made 4 or more searches.

Drum roll…

So which design resulted in a larger percentage of conversions? Well, neither.

Here’s the results from A/Bingo:

In other words, 167 out of 1006 visitors who saw the dark design performed 4 or more searches and 160 out of 1002 who saw the light did the same. The result is, as A/Bingo notes, statistically insignificant.

But there’s more to it than conversion rates

The difference was insignificant so by the what-percent-of-people-perform-for-four-or-more-searches metric, it wouldn’t matter which design I chose.

Ultimately, I stuck with the dark design for three very subjective reasons:

Despite a few folks not liking it, I’ve gotten a lot of feedback that people do like it too.

Almost as as importantly, I like it. The more I like the product I am building, the more likely I am to continue building it.

Everyone has a light site. Light is boring. Dark gives the site character (like I said, very subjective).

A few what-if’s to think about

What if the color scheme doesn’t make a difference on how many people make four searches, but it has a big difference on the percentage of people who make it to ten or twenty searches? Maybe the bright green’s effect is significant, but it takes longer for the difference to materialize.

What if the color scheme affects the percentage of people who buy a domain through Lean Domain Search? Maybe the bright green excites people more on average and they are more likely to buy.

What is the color scheme affects how long people spend on the search results page? Maybe people spend a significantly longer time on the page with the light background. What would that mean?

What if people are more likely to share Lean Domain Search because the dark color scheme makes it stand out?

What if changing the shade of green on the dark design produced a significant increase in the number of searches people make? (Did you know that Google A/B tested the affect of 41 different shades of blue? True story.)

What if the affect of introducing a site-wide design change for existing users (half of which open the site and have it gone from black to white) frustrates those users and makes them less likely to come back to the site?

What if measuring the number of searches is a poor metric? What if people make more searches because they haven’t found a domain name that they like?

I say this to make it clear that you should take this result — and any result — with a grain of salt. There are often second and third order effects of your changes and you need to think through them to make sure you didn’t miss something important. As Daniel Tenner notes, data is dangerous.

If you enjoyed this post, you should follow me on Twitter: @mhmazur. And check out Lean Domain Search, the best damn domain search engine ever.