How Optimizely Shrunk Google’s Market Share by 92%

Back in the mid-2000s, it was pretty difficult to gain a quantitative understanding of how your marketing efforts were working out. The only real A/B testing tool available was Google Website Optimizer.

It looked like this:

The process was painful. You had to take your page, think about all the different sections you wanted to A/B test, and put in script tags. For each test you wanted to run, you had to create a new page. And there was no way of easily visualizing all of this information—understanding the results of your tests was an exercise in frustration.

For your average marketer, it was practically impossible. Website Optimizer had been built by engineers for engineers, while the people who actually needed to be running A/B tests were left scratching their heads.

Then in 2010, Optimizely launched. By thinking deeply about who needed to do this job rather than just what needed to be done, then designing a tool specifically for that market, Optimizely revolutionized A/B testing on the web. And all they had to do was take out steps.

Tuning the Desire Machine

Evan Williams, one of Twitter’s co-founders, gives his secret to building a billion-dollar internet company:

“Take a human desire, preferably one that has been around for a really long time. Identify that desire and use modern technology to take out steps.”

The desires that Williams describes aren’t about summoning a car or a side of fries with a tap on your phone. They’re more basic and immutable: getting from point A to point B, or just eating food. We want what we’ve always wanted. What technology allows us to do is get it faster and without having to think about it.

Reducing steps requires some of the deepest thinking because it requires you to design for simplicity, rather than adding complexity. It’s about breaking problems down to their most basic form.

The best ideas come from deeply thinking about how to make things easier for other people, and help them accomplish their goals with the minimum amount of effort. You might get there with tech, but that’s also the part you should think about last.

11 Steps — Google Website Optimizer

Setting up a single A/B test with Google Website Optimizer was a long, nontrivial process. Here are the 11 steps Google outlined for setting up an experiment:

Name Experiment and Identify Pages:

1) Name your experiment. For example: Sign-up Form AB Test
2) Set your original test page URL: http://www.mysite.com/sign-up.html
3) Set your first variation test page URL: http://www.mysite.com/sign-up-b.html
4) Set your (optional) second variation test page URL: http://www.mysite.com/sign-up-c.html
5) Set your conversion page URL: http://www.mysite.com/thank-you.html

Install and Validate JavaScript Tags for:

Original test page:

6) Add the Control Script at the top of the page
7) Add the Tracking Script at the bottom of the page

First Variation test page:

8) Add the Tracking Script at the bottom of the page

Second variation test page:

9) Add the Tracking Script at the bottom of the page

Conversion page:

10) Add the Conversion Script

A/B Experiment Set-up: Preview and Start Experiment

11) Review the data, test the pages using the “preview” function, and then launch your test

And that was only if you were trying to run a single test on a static web page. If you wanted to add different variations, you had to keep poring through code. If you had a dynamic page, you really had to hack it to work. At each step of the way, because you were altering the code of every page on your web site, there was a chance that you’d introduce an error and have to start over.

Sean Ellis, who was VP of Marketing at Logmein at the time, wrote:

“One way I have worked around my engineering deficiencies has been to hire the skills onto the marketing team. For example, in my last long-term VP Marketing role I hired a front-end designer/engineer to design and code landing pages and a dedicated DBA to build reports and run ad hoc queries.”

Marketers like Sean Ellis who could acquire great engineering talent could run more tests, learn faster, and grow. Most marketers however, especially at startups, simply didn’t have the resources to accomplish this.

3 Steps — Optimizely

In 2010, former Google product managers Dan Siroker and Pete Koonen launched Optimizely—their deeply-thought solution to this problem. Optimizely couldn’t run more powerful tests than Google, but it did provide a graphic interface to run and manage the tests.

All marketers had to do was install a single line of Javascript on their site. That line generated a simple WYSIWYG editor that let them point and click to pick the site elements they wanted to test. Results were easy to access and learn from.

Optimizely cut GWO’s 11-step process into three short steps.

1. Insert Optimizely Javascript snippet in the head tag of your site.
2. Open up the visual editor, and click on each section of the site you want to test.
3. For each variation, click “add variation,” then click start experiment to preview and launch test.

The number of steps it took to A/B test—from turning to an engineer, pointing out the elements of your web page that you wanted to test, and having them insert script tags for each—was drastically cut down.

Optimizely thought deeply about the problem, and solved some of the hard technical problems behind building a WYSIWYG.

When they launched in 2010, Google Website Optimizer was all but abandoned. Marketers were suddenly able to run many types of experiments on their own, and all the A/B testing tools you see today solve the problem in the way that Optimizely did first. Optimizely took over the A/B testing marketing, and six years later, they still have the lion’s share.

How to Build the Habit for Deep Thinking

It would have been easy to improve on Website Optimizer by adding bells and whistles. Optimizely could have built a more powerful or more feature-rich version of the tool everyone knew. But by thinking deeply, Optimizely was actually able to cut features, make it less powerful, and create a far more useful tool in the process.

When we hear about a problem, our instinct is to jump in with a bunch of different solutions. Far more often than not, these are scattershot fixes that don’t get at the root causes of the problem. They do more harm than help.

Thinking deeply about reducing steps isn’t something that comes naturally to us—it takes practice and time.

• Ask the opposite. When building product, the internal biases and assumptions you hold often obscure the real problem. When you come up with a hypothesis about a product—or really anything—ask yourself, “What if the opposite were true?” Force yourself to keep to this line of reasoning for an hour and challenge each assumption you make.
• Storyboard the steps. Create a chart of the problem your problem’s trying to solve, and write out each step—no matter how large or small—it currently takes for them to solve it. Look for high-friction areas where you can make things easier and more efficient.
• Create a concept car. Optimizely takes the idea of the “concept car” from auto production to describe prototypes of a futuristic looking thing, only with software. Instead of thinking about your limitations, think deeply about what the future could look like. This is another way of working backwards. It allows you to build from the end-goal out.

Building the habit of thinking deeply starts with baby steps.