How would an evolving website work?

As a web developer who is also interested in A/B testing and evolution, it occurred to me that it would be fascinating to try to build a website that optimizes itself. I’ve been kicking around this idea for a while and wanted to share a few thoughts on it here to get feedback from anyone else that finds the idea promising.

Here’s the idea:

In traditional A/B testing, you specify multiple variations of some aspect of your site, measure which variation performs the best, then make the winner the default, and repeat. The types of things you test can range from simple headline and button color tests to complex tests that affect the user’s entire experience with your site.

In all of these cases though you have to figure out what variations you want to test. If you’re testing the headline, you need to specify which headlines you want to test. If you’re testing button color, you need to specify which colors to test, etc.

In the natural world, we see stunningly complex and optimized life forms that evolved little by little over billions of years. Evolution is similar to A/B testing in a sense, except that in evolution the variations happen by accident through genetic copying errors. Most of those mutations decrease the organism’s odds of reproducing, but occasionally they confer a benefit that causes that organism to have a better chance at surviving and reproducing. When that happens, the mutation gets passed on and may eventually spread to the rest of species over time. That “simple” process is what has led to all of the variety of life on earth.

Optimizing a web page through evolution poses many issues though.

How do you algorithmically mutate something on the page? Could you write a function that generates variations of a headline? Maybe. Would those variations be any good? Would you trust it enough to deploy into production?

I bet by analyzing tens of thousands of webpages, you could algorithmically identify common headline wording structures. Then maybe you could write code that intelligently customizes those headlines to your service.

You might be able to do the same with design. If you tried to categorize hundreds of homepage layouts, I expect you’d probably wind up with 20-30 common layouts that cover 90%+ of websites. Could you then write an algorithm that automatically tweaks your site’s layout to test these different layouts on your visitors? Could you do the same with color schemes? Maybe.

There’s also the problem of statistical significance. Running a simple two variation A/B test can take a long time if you don’t get a lot of traffic. Trying to get significant results for lots of algorithmically generated headlines might take forever. But maybe there are ways around that like the multi-armed bandit algorithm.

To sum it up, I think you’d need the following: a way to intelligently generate mutations on headlines, buttons, layout, etc + a ton of traffic or some novel method for picking the best variations + an organization that is willing to try such an experiment. It would not be easy, but I think it’s possible.

Imagine if it did work. Maybe your homepage converts users into sign ups at 10% now. You flip a switch, and in 6 months it increases your conversion rate 30% without any intervention on your part.

It would be revolutionary… if it worked.

The State of Emergent Mind

In 2014 I worked on a side project called Emergent Mind to learn more about AI, artificial life and related topics. The result was 10 interactive JavaScript visualizations ranging from Boids to neural networks.

I really enjoyed working on it and received a lot of great feedback, but I had to put it on hold after the arrival of my son last fall. I’d say the projects took about 10 hours each on average so they weren’t that time consuming, but with a newborn, a fulltime job and other obligations and interests I had to put it on the backburner for a while.

I’ve been weighing in my mind whether or not to resume work on it.

Here’s my thought process:

The Pros

  • AI and especially artificial life are fascinating to me. The fact that we are all here now is a testament to the incredible power of emergence and evolution. If I won the lottery and never had to work again, I’d probably spend a good chunk of my time working on projects to explore these areas.
  • AI is playing a larger and ever-increasing importance in our lives. I think we’re going to see an explosion of AI-enabled products and services over the coming decades… hopefully in a good way. Devoting time to learning it now will position me well to play a part in that future.
  • My impression is that AI researchers don’t tend to work on products and product people don’t tend to work on AI so there are probably a lot of opportunies at the intersection of the two.

The Cons

  • It’s hard. These 10 projects barely scratched the surface. My backpropagation post is at about the limit of my current math skills. I might (and that’s a big might) be able to teach myself, but flipping through PDFs of machine learning texts like The Elements of Statistical Learning is pretty daunting. I could go back to school, maybe get a masters in machine learning, but that brings me to the next point…
  • There are huge opportunity costs. Every hour I spend learning and working on AI projects is one less hour I get to spend learning about growth, JavaScript, spending time with my wife and son, relaxing, etc. Given a choice between going from an intermediate JavaScript developer to an expert with a few months of work and going from a beginner at AI to an expert in 5-10 years of work it’s hard to justify the latter.
  • No one can say for sure how AI will play out, but the possibilities have a lot of smart people very concerned. Read this WaitButWhy post for a great introduction and Superintelligence by Nick Bostrom if you want to learn more.
  • I love building products, but just don’t have many practical applications in mind for this type of work. The state of the art machine learning work seems largely applied to things like computer vision (boring to me), self-driving cars (interesting, but hard to work on from my home office), increasing the accuracy of various prediction techniques (somewhat interesting to me, but still no product in mind) and virtual assistants (interesting, but seems hard to compete with Facebook, Microsoft, Google, Apple, etc). It might just be a lack of imagination, but I can’t think of many products at the intersection of web app development and AI that excite me. If I were to build something it would probably wind up being a product like The Grid, though calling that AI seems more like a marketing move than actual AI (what is actual AI though?).

So, for now anyway, I think Emergent Mind will continue to stay on the backburner. Maybe one day I’ll pick it up again. We’ll see.

I’d love to hear your thoughts.