A/B test your headline without a data team

Your headline is the first thing almost everyone reads. Change it and you can change how many people scroll, click and buy. The trouble is knowing whether a new one is actually better, or whether you just had a good week.

An A/B test answers that. Half your visitors see the current headline, half see the new one, and you compare what they do. You don’t need a data team. You need a few rules, and a little patience.

The rules that keep a test honest

1. Change one thing. If you change the headline, the photo and the price at once, you won’t know which one did it.
2. Decide what counts as a win before you start. A sale, a signup, a click on Buy. Write it down. Picking whichever number looks best afterwards is how people fool themselves.
3. Show each person the same version every time. Someone who sees A on Monday and B on Tuesday muddies both.
4. Run whole weeks. People behave differently on a Saturday than on a Tuesday. Seven days is the minimum.
5. Don’t stop the moment one pulls ahead. Early leads come and go by chance. Know how many visitors you need before you start, and wait for them.

How many visitors you need

This is the part that surprises people. It depends on how often visitors do the thing today, and how big a difference you want to be able to spot.

Here’s what the standard calculation gives for two versions, at the usual 5% significance and 80% power (example rates):

• If 3% of visitors buy, spotting a 20% improvement takes about 13,900 visitors per version.
• If 10% of visitors click your Buy button, spotting a 20% improvement takes about 3,800 per version.
• If 3% of visitors buy and you’re looking for a 50% improvement, it’s about 2,500 per version.

Two practical lessons fall out of that. If your site is small, test bold changes rather than small tweaks, because big differences show up sooner. And consider measuring an earlier step, like clicks on Buy, which happens more often than a purchase and needs fewer visitors to read.

Write two headlines worth testing

A test is only as good as the idea behind it. Instead of swapping one word, try a different angle. For example:

• A, what it is: “Lightroom presets for moody film photos”
• B, what it does for you: “Make your phone photos look like film in one tap”

Other angles worth a test: who it’s for, the problem it solves, a number (“40 presets, one price”), or the result people get.

Set it up

You need three things: a way to split visitors, a way to keep each one on the same version, and a count of what each group does. odo does all three with a couple of attributes on your page. Put the current headline first and the new one right after it, hidden:

<h1 data-odo-experiment="Headline" data-odo-variant="control">Lightroom presets for moody film photos</h1>
<h1 data-odo-experiment="Headline" data-odo-variant="one-tap" hidden>Make your phone photos look like film in one tap</h1>

odo picks a version for each visitor, shows it, keeps the other hidden, and remembers the choice, so they see the same headline on every visit. If odo’s script can’t run, everyone sees the first one, which is why the current headline goes first.

If the headline comes from your own code, ask odo which version to show:

<script>
  const headline = window.odo && window.odo.experiment
    ? window.odo.experiment('Headline', ['control', 'one-tap'])
    : 'control';
</script>

If that code runs before odo’s script has loaded, paste odo’s early snippet, from your website’s settings in odo, into your page’s head first. Then it gets each visitor’s version from the very first line.

To check each version before anyone else sees it, open your page with ?odo_variant=Headline:one-tap on the end. That shows the version to you without counting it.

Two things to know. Experiments need the visitor id odo keeps, so on a site running in cookieless mode, and for visitors whose browser sends Global Privacy Control, everyone gets the first version and nothing is counted. And experiments are recorded along with clicks and forms, so keep that setting on for your website. It’s on by default.

Read the result without fooling yourself

When the test has run, odo compares each version with the control and gives a plain verdict:

• Winner or loser: the difference is clear enough to act on.
• Leading: one is ahead, but not clearly enough yet. Keep it running.
• No difference: with enough people in, the versions perform about the same.
• Too early: under a week, or too few people to say.
• Broken: something is wrong with the split itself.

odo calls nothing in the first seven days. After that, until each version has had the visitors it needs to spot a 20% lift, it only calls a difference that’s very clear. So checking every day doesn’t trick you into a false winner.

It also checks the split. If one version got far more visitors than the other, beyond what chance explains, something is off: a redirect, caching, a crash. The verdict says so, instead of handing you a number you can’t trust.

When you’re measuring sales, you also get revenue per visitor for each version, so a headline that sells fewer but bigger orders isn’t mistaken for a loser.

You can see the result in odo, or ask Claude or ChatGPT to review all your experiments through odo’s connection for AI assistants: here’s how.

Then finish it

Once there’s a verdict, ship the winner and delete the other version along with the experiment attributes. Old test code is easy to forget and confusing to find later. Then pick the next thing to test.

Not sure what that should be? Find the step where most people leave first: find where buyers drop off.