How to do a simple A/B Test

In digital marketing we often want to test to see what works best. What is most attractive to a potential customer? What message resonates best? What advertising campaign is most likely to get attention? How can we maximize the completion rate on a website landing page?

These kinds of questions often involve a simple test of two competing ideas or approaches. For example, we might have two banner advertisements or two registration pages, and we want to know which results in the most click-throughs or which is most likely to result in more leads. This is called an A/B test. Here’s how to do one right and not have to worry about the stats too much.

What is an A/B Test?
In marketing, an A/B test is a simple way to understand if there’s a meaningful difference between two competing approaches to a problem, such as a banner advertisement or a direct mail campaign. Using the test, we can measure the results of the two approaches and decide if there’s a real (statistically significant) difference.

A/B testing isn’t a new idea. I can remember decades ago talking to an executive at AMEX where they performed extremely elaborate versions of A/B testing on their direct mail campaigns supporting new credit card applications. They were looking for tiny differences in response rates across millions of mailings, and tested everything down to the color of ink. Usually we’re not nearly as sophisticated but the same principles apply.

How do I set up the Test?
Setting up the test isn’t that hard. Normally, you’ll have two approaches to a problem and a desired outcome. Here’s a few examples:

• Your agency comes back with a new creative idea for a banner ad, and you'd like to test it against an existing design. You want to see if the new creative idea gets more click throughs over the other.
• You have a new layouts for a web page. Which keeps people on the page the longest, the new idea or the old one you already have in place?
• You have an email newsletter. You want to test a different subject line to see which gets the best open rate
• You’ve a new direct mail campaign, but can’t agree whether to feature the discount offer again on the envelope, or picture of the new product. Which gets the best response to your order hotline?

In each case, you have a couple of alternative approaches to a problem, and an obvious way of measuring (counting) which is best.

What else should I know to make sure the test will work?

Here’s a few things to be careful about:

•  Ideally, make sure you’re actually measuring the right thing. For example, if you have two banner ads, and each features a different offer, different graphics and is placed on different websites – well, it’s really hard to know what is affecting click-throughs. Is it the websites that make a difference, or the offer or the graphic? The test will not tell you the answer. As much as possible, try and keep everything constant across each test except the specific thing you’re testing.  
• Try and randomize as much as possible. What does this mean? Here’s some examples:
•  If you’re testing two direct mail pieces across a sample of your database (say, 500 people), then make sure to randomly select people from the list and randomly place them in each group. This way, the selection process can’t influence the outcome.
•  If you’re running two different banner ads on a website to see which performs best, try and make sure they’re seen randomly, or as near as you can get.
• Try and make sure that you’re testing across a big enough sample. Bigger is usually better. Statistically, the power of the A/B test is related to the amount of data you gather, and hence the sample size. And remember, your data is often measured in terms of response rates, which in marketing can be very low. There’s no magic number, but try and get large.

How do I perform the test, and do I need to understand statistics?

Most people who do A/B testing never perform an actual test – they “eyeball” the data instead. As we’ll see, this can be a big mistake. It’s absolutely worth doing the tests because it’s easy to come to a wrong conclusion. The test involves statistics, but luckily for us there are online tools that do all the math for you. Here’s two examples:

Example One: Banner Advertising

You run two banner ads on the same website page that appear randomly to viewers (Banner A is old, and you think Banner B might do better). You use 5,000 impressions, 2,500 of each banner, and here’s the results:

	Impressions	Click Thrus
Banner A	2,500	25 (1%)
Banner B	2,500	32 (1.28%)

Eyeball this data, and most people would conclude that Banner B is easily better than Banner A. So let’s run a test to see if this is statistically the case – I’m going to use a simple tool supplied by the good people at Optimizer. (The tool is designed for a specific example for a website registration page, but we can easily adapt it for our use.)

To use the tool, let Control = Banner A and Variation = Banner B (remember, in this example you’ve a hunch that B is better than A). The “Number of Visitors” can equal the number of impressions, which is 2,500 in each case. The number of Conversions is the number of click-thrus for each banner (25 and 32 in our example).

Ignore the “p-value” unless you’re a stats geek. Hit Calculate and the tool will tell you if the difference in click throughs is really statistically significant. And the answer is – NO!

How can this be??? Surely, if there’s a difference in the results, it must mean our Banner B is better than Banner A!

Well, imagine you’re flipping a coin, and you flip it 100 times – assuming the coin is fair, you should get 50 heads and 50 tails, but you wouldn’t be surprised if you got 48 heads and 52 tails. Or even 45 heads and 55 tails. The same random chance with coin flips can influence our simple A/B test, so even though there’s a small difference in results, we can’t be certain this isn’t just random. You could be making a mistake going with Banner B. This is why it is so important to actually do a test – especially in situations where “responses” are very low, as is usual in most marketing activities.

Example Two: Website registration

You now run a test of two landing pages where visitors are asked to register. Page A is the old page you’ve used for a long time, and Page B is “variation” you want to test. You want to see which gets the most registrations, and here’s the data:

	Impressions/Visitors	Registrations
Landing Page A	13,000	83 (0.64%)
Landing Page B	8,900	78 (0.87%)

Plug in the numbers and… the answer is YES! But what does this mean?

It means, Page B performs better than Page A, within the margin of error of the statistical test. Let’s explore this.

In this case, we should note that the number of visitors on each page is very different (which is not to say that they aren’t random across each page, we assume they are. It’s pretty common to have different numbers given the practical realities of market research). Page A has 13,000 visitors, compared to only 8,900 on Page B. Page B has a better registration rate of 0.87% compared to 0.64% for Page A (the absolute numbers of registrations are very close, but are misleading). Eyeballing the data could be confusing, but the test makes it clear that Page B is the better bet.

Happy testing.

What’s at the core of Apple’s social media strategy?

Get into a discussion on marketing strategy, or attend any meeting with “brand” in the title, and it’s usually only a few nanoseconds before somebody invokes Apple and rattles on about how we need to be more like them and do exactly what they do. You can work for any kind of company, from a biotech start-up to a manufacturer of concrete, and this rule applies: Apple is the gold standard on how to do marketing right and create a winning brand that has a gravitational attraction for customers and a magic ability to mint cash.

Yet, when it comes to social media, Apple is doing exactly the opposite of everyone else.  Lemming like, most companies have leapt head-first into the social world, and most experts agree that having a solid social strategy is critical for building a successful brand. Yet Apple is largely absent from social. So, how come the most successful brand on the planet has been such a contrarian when it comes to social media, and been so successful doing it?

‘Reputation management‘ is often the reason given for an active social program, especially for established brands like Apple, the logic being that reputations are made or lost amidst the babble of customers, prospects and competitors. But the linkage between reputation, brand, and company performance is fluid, with one not clearly being dependent on another. Indeed, there’s plenty of crappy companies that have award-winning social programs  - Kodak comes to mind, or Dell.

Apple’s attitude to social is more likely bound-up with their fetish for control. They control the message and they control the user experience. They close others out and control the supply chain. They operate on a principle that information scarcity will drive anticipation and demand. None of this would argue in favor of an aggressive social program, where control is relinquished and being closed just doesn’t work. Nevertheless, it’s hard to argue against the success of Apple’s approach, at least under Jobs’ tenure.

This approach wouldn’t work for most companies.  At a practical level, most of us don’t have the discipline to make it work. But more importantly, ignoring social won’t work because, sadly,  none of us actually are Apple.  Until recently, Apple was the most valuable company in the world. They command attention, people actively listen to the slightest rumor from them, the media fawn. You and I need to fight for these things. It’s the difference between playing offense and defense.

But there’s a bigger lesson to be taken from this.

Apple is good at many, many things, among them having a very clearly articulated business strategy that permeates everything they do. Their approach to social is an articulation of this strategy – it just so happens that this means they do very little social at all. Too often, social is seen as an end in itself, and having a social media strategy as something distinct from broader business objectives. At its core, that’s a recipe for failure.

The Crowdsourcing Myth and the White House

A while back the Obama administration, in a wild fit of optimism, created We the People, an online forum where any citizen can petition the government on a question, and if sufficient people sign the petition they’ll get a response. Great idea, right? Government needs to be more open and responsive, and this is open to all. It’s a form of crowdsourcing, where citizens with common cause can directly engage with elected officials.

Turns out, the idea is great in theory and absurd in practice, much like crowdsourcing itself. Within weeks of launching We the People petitioners from 50 states were asking to secede from the union, and a petition to build a Star Wars’ Death Star attracted over 34,000 signatures (but was regrettably turned down). The Brit CNN talk show host Piers Morgan, a fierce gun control advocate, was quickly the subject of a petition to have him deported. In response to all this nonsense the White House raised the threshold for getting a response from 25,000 to 100,000 signatures, a move only likely to encourage the crackpots and fringe dwellers even further. The UK governments’ e-petitions site has a 100,000 signature threshold, and still gets petitions attempting to save chocolate bars.

Opening the opinion floodgates is admirable, but we forget that those of us with moderate views are labeled the “silent majority” for a good reason. It’s not that we’re lazy, though that could be true, but that the frenzied extremists are the people who are always going to exert themselves the most. If you doubt me, take a look at the comments on any online coverage about the Obama administrations recent efforts to enact gun control legislation – the ferocity of opinion is itself a great argument for keeping kitchen utensils out of the hands of most Americans, never mind semi-automatic firearms.

Crowdsourcing is a dangerous myth. The wisdom of the crowd is far too easily drowned out by the madness of the masses.

US Election Costs $50 Per Vote –Money Well Spent?

The presidential election of 2012 may not be remembered for who won or lost, or the issues, or the storm that wrecked the east coast: There’s a good chance it will be remembered by how much it cost.

By some estimates, over $6 billion (with a ‘B’) has been spent collectively on the various election races nationwide, with something close to a billion being spent on the presidential election alone. With at least 120 million Americans expected to go to the ballot, that’s a staggering $50 per vote. This compares with under $20 per vote in 2000.

This is unquestionably great news if you’re a TV station owner, especially if you’re Ohioan, but of dubious merit for just about everyone else and for democracy at large. But let’s ignore the Big Question about the wisdom of mixing all that money in a democratic election process. I have a more fundamental question: Is this money well spent? Said another way, can you buy an election?

There’s no question that advertising is persuasive to some degree, but can it influence voting? Not so much. Stephen Dunbar at Freaknomics  finds that campaigns managers and insiders from all political parties don’t see a nice linear relationship between spending and outcome, but they recognize it contributes. While researchers acknowledge that the biggest spender is also most often the winner, when all other factors are taken into account, the spending isn’t the root cause of success. To cite recent examples, Meg Whitman in California or McMahon in Connecticut each spent a considerable part of their private fortunes on loosing campaigns. So, the good news is you apparently can’t buy an election.

But wait. There has to be a reason all this money is sloshing around in politics – why are we all subjected to so many crappy TV ads unless they do something?

Advertising does a couple of things very well. First, and most obviously, advertising raises awareness. In this regard, political candidates are no different to dog food or Viagra, at least from an advertising perspective (there may be other similarities which I won’t pursue). This is why so many ‘third party’ candidates rail against the political system – they aren’t able to compete for awareness is a noisy, cluttered political environment. (Indeed, many Americans are surprised and confused to find that there’s actually four candidates on the ballot for president this year. Most voters have never heard of two of them).

The second effect of advertising is to set the agenda. The famous adage about the news business –  news organizations can’t tell us what to think, but they can tell us what to think about – is also true of advertising. This election cycle, the Democrats did a masterful job of framing the political debate, so that the focus became less about the economy and more about social, environmental and health issues. This didn't directly persuade voters to vote for any specific candidate, but it did provide evaluation criteria that favors one over the other.

Advertising can have others effects, especially for so-called ‘low involvement’ buying decisions (think breakfast cereal or washing detergent), but for most situations it’s influence is weak, indirect and fleeting.

As goes political advertising, so goes advertising everywhere.  Those we wish to influence in business may not be heavily swayed by our own advertising and self-promotion. We can make them aware of our company, we can raise issues we think are significant and warrant attention – but we will need to rely on other tools to really influence decision making.

The Value of Social Media

Ever since the Facebook IPO debacle last May there’s been a lot of fretting about the real value of social media platforms. Facebook’s stock opened at $38 a share, and today trades at about $21, a loss of 45% or about $25 billion dollars, or the equivalent of the GDP of a small country – say Cyprus or Panama.

Of course not all social platforms have disappointed so magnificently. LinkedIn, the social professional networking platform, IPO’d in May 2011, and first day investors have made a healthy 50% return. There are other success stories.

The valuation of social platforms isn’t just about investor returns. We need to consider the value proposition to users and to those that will provide income to these platforms – potential sponsors. Most social platforms have at their core an advertising model for making money, and advertisers have also begun to express their value judgments. Responses have been mixed. Back in May this year, shortly after Facebook’s roller-coaster IPO, GM announced that it was withdrawing advertising support, citing poor performance. Many other companies have taken a measured approach to all online advertising, although according to Forrester, advertising revenues for social platforms are expected to rise by 34% through 2014. To some extent, advertising on social is a victim of its own success: clutter and saturation make any advertising or sponsorship difficult. And many potential sponsors see an opportunity to disintermediate all media, and reach out directly to potential customers with their own tailored content. Then there’s the Transparent ROI Problem – online is so amenable to measurement, smart companies are able to precisely gage the hard returns on investments (or lack thereof), and increasingly are discounting softer brand benefits.

One thing is certain – the value proposition to users of social is beyond compelling. For most teens, Facebook is as necessary to life as oxygen. For most everybody else, social is an ingrained and everyday part of life, like the morning cup of coffee or, dare I say it, like the daily newspaper used to be. It is this collective addiction to social media that is fuelling a social media investment bubble, but until we find a clearer conjunction of shared value between users, sponsors and the platforms, the real potential won’t be realized.

To some extent, LinkedIn shows the way and for sponsors and users, the meeting point for shared value is bound up in the idea of community. This is where we find the common values of like-minded users of a product, or passionate followers of a brand. In LinkedIn's case, they attract professionals who want a mediated way to network with colleagues and companies, and businesses want a way to find talent. Everyone sees value. Today, too many social platforms are hoping to exploit business value by mining users’ personal information, marketing this data to companies. Trouble is, users don’t see the shared value – they see exploitation, and that’s not good news for social media companies.

The Emperor Wants Clothes

We used to have an assumption that is the digital world we remained cloaked – almost anonymous – unless we choose to reveal ourselves, and even then we were able to invent a self of our choosing. Not anymore.

As I’ve mentioned in a previous post, everyone now knows that as we rummage around in the virtual world, our digital trail is soon followed: What we do, where we go, who we are, and what we think can all be discovered and refactored with unnerving ease. Our online selves are laid bare.

Us marketing types are very happy with this situation. We like naked consumers who cavort online as if well-dressed, because this unrequited intimacy allows us to target them very effectively. Knowledge is always powerful, and in marketing circles the manifold details we can gather online make us giddy with excitement about how we can tailor loving entreaties the better to woo prospects.

The whole thing is a parody of the old children’s story of the Emperor’s Clothes, expect that in this version the Emperor is starting to demand we give him his old wardrobe back.

Punters know they’re being digitally stripped searched and they’re not thrilled.  They want some protection, some dignity, some rights. Enter Do Not Track. As early as 2007 the US Federal Trade Commission was approached about creating a “do not track” list, similar to the “do not call” lists that exists to suppress telephone solicitations. Over time, a more practical, technical solution has become favored: The implementation of a HTML header field that automatically signals a user’s willingness to allow tracking. This is a small step in the inevitable direction of giving consumers more rights to privacy, but it may not be enough to placate regulatory bodies like the FTC or EU regulators.

So, what do we marketing types do when the Emperor has robes again? More permission-based marketing is one outcome. A reliance on cultivating trust and a relationship with prospects is another. Certainly, the crutch of unfettered access to consumer information may soon be gone.

The Gladwell Effect

A virulent meme of the moment is 10,000 Hours – the idea, popularized by Malcolm Gladwell in his book Outliers, that brute force persistency is the route to greatness.

 In his best-selling book, Gladwell examines the factors that contribute to high levels of success and acclaim. Looking at everyone from Robert Oppenheimer to Bill Gates, he concludes that success is largely predicated on having the stamina and determination to work at a task for a total of around 10,000 hours – what he calls the 10,000-Hour Rule.

Really?

One of his examples is The Beatles, who Gladwell points out spent much of their early days in a daze, in Hamburg, playing and playing and playing. Having read about their exploits in Germany, it’s actually amazing they even survived the experience, never mind rose to acclaim. But to say that the critical ingredient that led to their fame was being on stage together for months on end is, at best, misleading. There’s just so much more to account for.

For one thing, The Beatles adroitly (or fortuitously) surrounded themselves with great talent. To take an example, their producer on most of their recordings, George Martin, was enormously influential on their musical development, making the studio an instrument in itself and pushing the band to explore more complex sounds. Then there’s Brian Epstein, who took their raw talent and turned it into a mop-haired product for worldwide consumption. Oh, and in case we forget, Lennon and McCartney wrote some pretty good tunes, a talent that transcends anything they might have picked-up at the Ratskeller.

Perseverance is undoubtedly a characteristic of greatness. So is its near-neighbor, obsession. I’d actually argue that the real driver here is passion, an ardor for what you do. But this is never enough and to argue otherwise is an oversimplification.

Gladwell can’t be entirely blamed – although some, like scientist Stephen Pinker, have claimed his whole argument is flawed. Gladwell’s idea is more nuanced, but the popular interpretation is a reduction to a direct cause-and-effect: If only we all tried harder, we’d be rock stars.

It’s a very human failing to try and account for all results by isolating a single variable. In marketing, we do this all the time – be it attempting to understand what led to a sale, why that video went viral, or what caused our competitor to beat us on a deal. It’s too easy to say it was all down to the salesman, or the clever script, or the fact we didn’t have that one specific feature in our product. Usual this reductive reasoning is all wrong.

In marketing, don’t expect simple answers. And don’t anticipate that repetition and persistence alone will drive success.