Sunday, March 10, 2013

How to do a simple A/B Test

In digital marketing we often want to test to see what works best. What is most attractive to a potential customer? What message resonates best? What advertising campaign is most likely to get attention? How can we maximize the completion rate on a website landing page?

These kinds of questions often involve a simple test of two competing ideas or approaches. For example, we might have two banner advertisements or two registration pages, and we want to know which results in the most click-throughs or which is most likely to result in more leads. This is called an A/B test. Here’s how to do one right and not have to worry about the stats too much.

What is an A/B Test?
In marketing, an A/B test is a simple way to understand if there’s a meaningful difference between two competing approaches to a problem, such as a banner advertisement or a direct mail campaign. Using the test, we can measure the results of the two approaches and decide if there’s a real (statistically significant) difference.

A/B testing isn’t a new idea. I can remember decades ago talking to an executive at AMEX where they performed extremely elaborate versions of A/B testing on their direct mail campaigns supporting new credit card applications. They were looking for tiny differences in response rates across millions of mailings, and tested everything down to the color of ink. Usually we’re not nearly as sophisticated but the same principles apply.

How do I set up the Test?
Setting up the test isn’t that hard. Normally, you’ll have two approaches to a problem and a desired outcome. Here’s a few examples:

  • Your agency comes back with a new creative idea for a banner ad, and you'd like to test it against an existing design. You want to see if the new creative idea gets more click throughs over the other.
  • You have a new layouts for a web page. Which keeps people on the page the longest, the new idea or the old one you already have in place?
  • You have an email newsletter. You want to test a different subject line to see which gets the best open rate
  • You’ve a new direct mail campaign, but can’t agree whether to feature the discount offer again on the envelope, or picture of the new product. Which gets the best response to your order hotline?

In each case, you have a couple of alternative approaches to a problem, and an obvious way of measuring (counting) which is best.

What else should I know to make sure the test will work?
Here’s a few things to be careful about:

  •  Ideally, make sure you’re actually measuring the right thing. For example, if you have two banner ads, and each features a different offer, different graphics and is placed on different websites – well, it’s really hard to know what is affecting click-throughs. Is it the websites that make a difference, or the offer or the graphic? The test will not tell you the answer. As much as possible, try and keep everything constant across each test except the specific thing you’re testing.  
  • Try and randomize as much as possible. What does this mean? Here’s some examples:
    •  If you’re testing two direct mail pieces across a sample of your database (say, 500 people), then make sure to randomly select people from the list and randomly place them in each group. This way, the selection process can’t influence the outcome.
    •  If you’re running two different banner ads on a website to see which performs best, try and make sure they’re seen randomly, or as near as you can get.
  • Try and make sure that you’re testing across a big enough sample. Bigger is usually better. Statistically, the power of the A/B test is related to the amount of data you gather, and hence the sample size. And remember, your data is often measured in terms of response rates, which in marketing can be very low. There’s no magic number, but try and get large.

How do I perform the test, and do I need to understand statistics?
Most people who do A/B testing never perform an actual test – they “eyeball” the data instead. As we’ll see, this can be a big mistake. It’s absolutely worth doing the tests because it’s easy to come to a wrong conclusion. The test involves statistics, but luckily for us there are online tools that do all the math for you. Here’s two examples:

Example One: Banner Advertising
You run two banner ads on the same website page that appear randomly to viewers (Banner A is old, and you think Banner B might do better). You use 5,000 impressions, 2,500 of each banner, and here’s the results:

Click Thrus
Banner A
25 (1%)
Banner B
32 (1.28%)

Eyeball this data, and most people would conclude that Banner B is easily better than Banner A. So let’s run a test to see if this is statistically the case – I’m going to use a simple tool supplied by the good people at Optimizer. (The tool is designed for a specific example for a website registration page, but we can easily adapt it for our use.)

To use the tool, let Control = Banner A and Variation = Banner B (remember, in this example you’ve a hunch that B is better than A). The “Number of Visitors” can equal the number of impressions, which is 2,500 in each case. The number of Conversions is the number of click-thrus for each banner (25 and 32 in our example).

Ignore the “p-value” unless you’re a stats geek. Hit Calculate and the tool will tell you if the difference in click throughs is really statistically significant. And the answer is – NO!

How can this be??? Surely, if there’s a difference in the results, it must mean our Banner B is better than Banner A!

Well, imagine you’re flipping a coin, and you flip it 100 times – assuming the coin is fair, you should get 50 heads and 50 tails, but you wouldn’t be surprised if you got 48 heads and 52 tails. Or even 45 heads and 55 tails. The same random chance with coin flips can influence our simple A/B test, so even though there’s a small difference in results, we can’t be certain this isn’t just random. You could be making a mistake going with Banner B. This is why it is so important to actually do a test – especially in situations where “responses” are very low, as is usual in most marketing activities.

Example Two: Website registration
You now run a test of two landing pages where visitors are asked to register. Page A is the old page you’ve used for a long time, and Page B is “variation” you want to test. You want to see which gets the most registrations, and here’s the data:

Landing Page A
83 (0.64%)
Landing Page B
78 (0.87%)

Plug in the numbers and… the answer is YES! But what does this mean?

It means, Page B performs better than Page A, within the margin of error of the statistical test. Let’s explore this.

In this case, we should note that the number of visitors on each page is very different (which is not to say that they aren’t random across each page, we assume they are. It’s pretty common to have different numbers given the practical realities of market research). Page A has 13,000 visitors, compared to only 8,900 on Page B. Page B has a better registration rate of 0.87% compared to 0.64% for Page A (the absolute numbers of registrations are very close, but are misleading). Eyeballing the data could be confusing, but the test makes it clear that Page B is the better bet.

Happy testing.

Monday, February 11, 2013

What’s at the core of Apple’s social media strategy?

Get into a discussion on marketing strategy, or attend any meeting with “brand” in the title, and it’s usually only a few nanoseconds before somebody invokes Apple and rattles on about how we need to be more like them and do exactly what they do. You can work for any kind of company, from a biotech start-up to a manufacturer of concrete, and this rule applies: Apple is the gold standard on how to do marketing right and create a winning brand that has a gravitational attraction for customers and a magic ability to mint cash.

Yet, when it comes to social media, Apple is doing exactly the opposite of everyone else.  Lemming like, most companies have leapt head-first into the social world, and most experts agree that having a solid social strategy is critical for building a successful brand. Yet Apple is largely absent from social. So, how come the most successful brand on the planet has been such a contrarian when it comes to social media, and been so successful doing it?

‘Reputation management‘ is often the reason given for an active social program, especially for established brands like Apple, the logic being that reputations are made or lost amidst the babble of customers, prospects and competitors. But the linkage between reputation, brand, and company performance is fluid, with one not clearly being dependent on another. Indeed, there’s plenty of crappy companies that have award-winning social programs  - Kodak comes to mind, or Dell.

Apple’s attitude to social is more likely bound-up with their fetish for control. They control the message and they control the user experience. They close others out and control the supply chain. They operate on a principle that information scarcity will drive anticipation and demand. None of this would argue in favor of an aggressive social program, where control is relinquished and being closed just doesn’t work. Nevertheless, it’s hard to argue against the success of Apple’s approach, at least under Jobs’ tenure.

This approach wouldn’t work for most companies.  At a practical level, most of us don’t have the discipline to make it work. But more importantly, ignoring social won’t work because, sadly,  none of us actually are Apple.  Until recently, Apple was the most valuable company in the world. They command attention, people actively listen to the slightest rumor from them, the media fawn. You and I need to fight for these things. It’s the difference between playing offense and defense.

But there’s a bigger lesson to be taken from this.

Apple is good at many, many things, among them having a very clearly articulated business strategy that permeates everything they do. Their approach to social is an articulation of this strategy – it just so happens that this means they do very little social at all. Too often, social is seen as an end in itself, and having a social media strategy as something distinct from broader business objectives. At its core, that’s a recipe for failure.

Saturday, January 19, 2013

The Crowdsourcing Myth and the White House

A while back the Obama administration, in a wild fit of optimism, created We the People, an online forum where any citizen can petition the government on a question, and if sufficient people sign the petition they’ll get a response. Great idea, right? Government needs to be more open and responsive, and this is open to all. It’s a form of crowdsourcing, where citizens with common cause can directly engage with elected officials.

Turns out, the idea is great in theory and absurd in practice, much like crowdsourcing itself. Within weeks of launching We the People petitioners from 50 states were asking to secede from the union, and a petition to build a Star Wars’ Death Star attracted over 34,000 signatures (but was regrettably turned down). The Brit CNN talk show host Piers Morgan, a fierce gun control advocate, was quickly the subject of a petition to have him deported. In response to all this nonsense the White House raised the threshold for getting a response from 25,000 to 100,000 signatures, a move only likely to encourage the crackpots and fringe dwellers even further. The UK governments’ e-petitions site has a 100,000 signature threshold, and still gets petitions attempting to save chocolate bars.

Opening the opinion floodgates is admirable, but we forget that those of us with moderate views are labeled the “silent majority” for a good reason. It’s not that we’re lazy, though that could be true, but that the frenzied extremists are the people who are always going to exert themselves the most. If you doubt me, take a look at the comments on any online coverage about the Obama administrations recent efforts to enact gun control legislation – the ferocity of opinion is itself a great argument for keeping kitchen utensils out of the hands of most Americans, never mind semi-automatic firearms.

Crowdsourcing is a dangerous myth. The wisdom of the crowd is far too easily drowned out by the madness of the masses.