It’s not what people say, it’s what they do

Never make management decisions for a website based on opinions. There is often a Jekyll and Hyde difference between what people say and what they do.

SIMS is a hugely popular simulation game. They wanted to improve sign up for optional registration for those who had purchased the game. At one stage they tested two sign up pages, one of which was 40 percent more successful. Now, 40 percent is a huge, massive difference.

I have shown hundreds of web professionals the two pages and asked them to choose what they thought was the most effective page. About 80 percent of people go for version B. This maps well to the voting that occurred on the site Which Test Won? (Well worth visiting if you’re interested in this sort of stuff.) The people who go to Which Test Won? tend to be in the business of managing websites.

The problem is that it was Version A that was actually 40 percent more successful. And this is far from a unique occurrence. To our eyes, Version B looks better, but the cold hard data shows that Version A massively outperforms it.

Our world is changing radically. The age of intuition, gut instinct, opinion, and natural creativity is on the wane. These are romantic notions and we love to believe in them, but when the data comes in they are being increasingly proven not just to be wrong, but horribly wrong.

The worst way to design a website is to get five smart people in a room drinking lattes and posting post-it notes. The longer you leave them the worse the website becomes. The next worst way is to get 10 customers in a room drinking lattes and giving their opinions on the new design. That model is really, truly broken.

The relentless march of cold, hard data can be seen everywhere. There is huge resistance to this evidence, but resistance is futile. Take poker. “Phil Hellmuth Jr. may be the world’s most decorated gambler,” TIME wrote in July 2010. “But last year it all began to fall apart. Hellmuth, 45, lost money and failed to make the final table of even one tournament for the first time in more than a decade.

“Was it his cards? No, Hellmuth says, pacing the floor of his suite at New York City’s Plaza Hotel. He blames the new breed of math nerd, those guys using a mountain of sortable data from the millions of hands played online to dominate the game. “The reason I won 11 bracelets is my ability to read opponents,” he explains. “These new guys are focused on the math. And they are changing everything.”

The world of human behaviour is much more observable and measurable today. Computers can crunch data like never before and we spend more and more time in the online world, a perfect place to be measured and observed.

I voted for Version B, the page that was much less successful. It’s scary. How can our opinions be so wrong so much of the time? But that’s not really the question. Rather, the question is: How can we use the data to make better decisions?

Time: Attack of The Math Brats

SIMS Online Game A/B Test


9 responses

  1. “Creativity is the ability to generate innovative ideas and manifest them from thought into reality. The process involves original thinking and then producing.” (Wikipedia) Lets hope it does not desert us soon.

  2. It is difficult to analyse this post because the page now only shows the successful Version A Sims image. However, their commentary suggests that the only difference between the versions was in the free offer:

    “Both pages had identical buttons and nav bars. The benefits for both offers were also identical, but the copy describing them differed. Version A’s headline emphasized the ‘Free Town’ offer whereas Version B’s headline emphasized the ‘Free Game Content’ offer. Images matched the narrative of the copy.”

    This isn’t really a case where you might expect a usability consultant to know one way or the other. Certainly not as much as someone who actually plays the game, so, it’s an ideal place to test.

    Assuming I haven’t missed anything, it seems to me that your article is sensationalist in relation to the Sims reference and plays down the role of the user experience expert in an unbalanced way.

  3. I agree with you Gerry. The WEB is defining new standards, nearly daily, and dynamically denouncing and debunking old standards as “broken”. Customer behaviour is capturable and measurable online. The WEB is a virtual “Freudian” paradise on earth, for behavioural analysists,to build the modern day equivalents of Charles Babbage’ difference engine and Alan Turing’s artificial intelligence engine. Empirical evidence based technologies are on the increase. i.e. recommendation engines, online Poker, search. The genie is out of the bottle and we are embarking on the cusp of new era ..

  4. Hi Olly. Yes, that site was useless not linking to the actual test. She writes a whole post on the results and doesn’t link to the test or show both screens. I wrote a comment on the site, so hopefully the owner will have a think about her readers.

    The test with both images can be found here:

    I think using the actual evidence is crucial to improving and evolving the online experience for our users. of course this has to be translated for more senior management, without losing the point in the process. HiPPOs tend to prefer gut feelings to facts.

  5. I second Michelle, the hardest part is always getting the HiPPOs to look at the evidence rather than make decisions based on their personal opinions. I have no problem with managers making decisions on sticking points, but I’d love to have all the time back I’ve wasted trying to get the exact colour the manager was thinking of while the glaring content problems remain unresolved.

  6. We designed parts of our site just as Gerry mentions, except more people and no lattes. I wager that no one who participated is happy with the end result. So much discussion and compromise watered down the efforts.
    This time around, we kept the group discussion to idea generation, then a small group– people who work with the site, usability/accessibility, and our customers regularly — worked out the final page and text. We haven’t lauchned it yet, but I think everyone will be much happier this time around.
    Also, what our customers tell us about how they use the page and what our usage stats tell us don’t match. Again illustrating that what they say and what they do are often different. We went with the data — and our gut.

  7. Renee, good luck with the new design-looks like you’re on the right track.

    Brian, I feel your pain. Some managers think about all the wrong things when it comes to the Web.

    Michelle–thanks very much for the right link. That was really my mistake; I should have checked out the link I placed more throughly. (Sorry about the poor link, Olly.

  8. A cracking post Gerry.

    I’ll give you an anecdotal from a government department.

    Designers build 2 big shiny colorful homepage options. They print them out on enormous A1 paper. The project manager shows the marketing manager the A1 sheets and says ‘which one do you like best’. The marketing manager, who is in a bad mood, sulkily points to his favourite without much thought. Job done.
    So no lattes, but not exactly evidence based. The final product did a good job of proving that you can fit a lot more on an A1 sheet than you can on a computer screen.

    I picked B, but then I didn’t realise A contained a picture of what was on offer (and that it would be appealing to people who knew the game).

    So I’d say that so long as the person choosing knows their subject matter and target audience their hunch may often be correct, or at least a good place to start.

    Finding a person who truly understands both the product and the audience plus something about marketing is near impossible of course - but the closer you get the better your starting point will be.
    Measuring results in a scientific and objective way is, as you say, a million times better than guessing. You do need someone capable of making the measurement and interpreting the results though I suppose.

  9. BIG: I’ve seen that movie too..
    RE: Dave’s point above: Isn’t this reminiscent of “BIG” with Tom Hanks all over again. i.e. Tom Hanks becomes a small boy in a man’s body who moves into the TOY marketing biz. He has the fastrack on what will sell, after all he is a boy . Selecting which toy will sell is a no-brainer for Hanks. Is there a moral here , is there a message, he knows what will sell. Steve Jobs knows what will sell…is he a boy in a man’s suit?

Leave a Reply