Experiments

An experiment shows each visitor one version of something (a headline, a price, a checkout button), keeps them on it, and counts who goes on to convert. odo says when one version is better for sure, and by how much, and acts on it.

There are two ways to run one, and odo reads and calls both:

• odo changes the page. odo’s script makes each version’s changes itself (new text, another link or image, something hidden or shown, a few styles), with nothing to deploy. See experiments odo runs.
• In your code. Your site shows the versions with odo.experiment() or data-odo-experiment markup, and you register it with odo under the same name, so odo plans, reads and calls it.

How it works#

• Each visitor gets one version (a variant, in code) and keeps it, on every page and every visit: it’s worked out from their odo_vid, so the same visitor always lands on the same one, whichever way the experiment runs.
• The first variant is the control, the original: what everyone gets before odo has run, where it’s blocked, and on cookieless sites. A variant named control is the control wherever it’s listed.
• A visitor counts once, in the variant they were shown first, and converting means reaching the outcome after being shown it.
• Names are letters, digits, spaces and _ - . :, up to 60 characters for an experiment and 40 for a variant.

Experiments need a visitor id, so a cookieless site, and visitors whose browser sends Global Privacy Control, always get the control and aren’t counted. They’re recorded with clicks and forms, so the site’s Clicks, forms, errors and page speed setting has to be on. odo won’t start an experiment on a site where either stops it, and says why.

In your code#

const variant = odo.experiment('Pricing', ['monthly first', 'yearly first']);
if (variant === 'yearly first') showYearlyFirst();

odo.experiment(name, variants, weights) returns this visitor’s variant. Weights are optional ([3, 1] sends three in four to the first). If your server already picked one, say which: odo.experiment('Pricing', 'yearly first') records it as shown.

Code that might run before odo’s script, or where it’s blocked, either uses the early snippet (then odo.experiment answers from the page’s first line) or checks first:

const variant = window.odo && window.odo.experiment ? window.odo.experiment('Hero', ['control', 'short']) : 'control';

In your HTML#

Write the control first, then each other version as its own element with hidden, all marked with the experiment’s and the variant’s name:

<h1 data-odo-experiment="Hero" data-odo-variant="control">Every sale, live</h1>
<h1 data-odo-experiment="Hero" data-odo-variant="short" hidden>Know what sells</h1>

odo shows each visitor theirs and keeps the others hidden, whatever your CSS says. Before it runs, the control is what shows, so nothing flickers in. Content your scripts build later is for odo.experiment.

Preview a variant#

Add ?odo_variant=Hero:short to a page’s address to see that variant on that load. Several go together, with commas: ?odo_variant=Hero:short,Pricing:B. Previews aren’t counted, and they work the same for experiments odo changes the page for.

Experiments odo runs#

odo runs experiments end to end: you, your server or your AI assistant say what to test, odo shows each visitor one version and keeps them on it, reads the result every 15 minutes, calls it, tells you, and can show the winner to everyone.

Make one#

On the Traffic dashboard, under Sales and goals, the Experiments card’s New (New experiment while it’s empty) opens the form:

• Name, and the Page it runs on: /pricing, /blog/* (* matches anything, slashes too), or * for every page. With several websites, which one.
• Judged by: Sales (a payment traced to the visitor), one of your goals, an event (Signup), or reaching a page (/thanks).
• Hypothesis, if you like: what you expect, and why.
• How it runs: odo changes the page, or In my code.
• Versions: 2 to 5, the original first. When odo changes the page, each other version lists its changes (up to 10). In your code, each has a key (exactly what your code calls it) and a few words on what it changes.
• When it’s called: Show the winner to everyone (when odo changes the page; on unless you turn it off) and Call it after 7 to 90 days (28 unless you choose).

Save as draft, or Start now. A draft can change freely. Once it has run, only its hypothesis, Show the winner to everyone and Call it after can, so what visitors saw stays what’s measured: to test something else, make a new one. Visitors are split evenly between versions, unless weights from the REST API or an assistant say otherwise.

The same can be done from a server with a key that has the Experiments permission (the REST API), or by an AI assistant you’ve let run experiments (below). Owners, admins and managers make and run them; shared views never show them.

What odo can change on a page#

Each change names its elements with a plain CSS selector of up to 200 characters: tags, .classes, #ids, [attributes], the combinators >, + and ~, and structural pseudo-classes such as :first-child, :nth-child() and :not(). Never the page itself (html, body), its head, or anything that loads or runs code (script, style, iframe, svg and the like). A change touches at most 20 elements.

There’s deliberately no way to set HTML, add a script or a stylesheet, set an event handler or any other attribute, or point a link or an image anywhere but https:// or your own site. Every change is checked when it’s saved, and again in the visitor’s browser before it’s made: one that fails is skipped.

Without flicker#

odo’s script reads what your site runs (cached for a minute) and loads a second small script, odo-x.js, only on sites where odo changes pages. A version is shown, and counted, once one of the elements it changes is on the page, so a single-page app’s later pages get theirs as they’re drawn. A shipped version’s changes go to every visitor, cookieless ones too, and count nobody.

The script keeps what it read for the next page, so for a returning visitor the changes are made the moment it runs. To keep the original from showing first, add the early snippet to your pages. On the pages an experiment runs on, it hides the elements experiments change from the first paint until odo has changed them, for 800 ms at most, for every returning visitor, whichever version they have (so every version’s visitors see the page alike). If the 800 ms run out first, that page keeps the original and counts nobody, rather than switching under the visitor’s eyes.

A visitor’s first page has nothing kept yet, so odo waits for what the site runs: if it comes within 800 ms it’s used on that page (the original may show for that moment), and otherwise from their next page. Serving odo from your own domain? Forward /js/odo-x.js and /js/x/ too (see ad blockers).

Statuses#

In your code, odo only reads and calls an experiment: what visitors see is up to your code, so pausing, stopping or shipping one there means changing the code too.

How odo calls it#

odo reads every running experiment every 15 minutes and acts only on its verdict, which won’t call a winner early (see how the verdict is called):

• Winner: decided. When odo changes the page, it shows the winner to everyone from then on (Shipped in odo), unless Show the winner to everyone is off: then everyone sees the original until you ship it. In your code, odo asks you to ship it, in a notification and a Next move.
• Loser, every version worse than the original: decided, and the original kept.
• No difference, once its plan is in: decided.
• Broken: never acted on. odo says why, at most once a week, and waits for a fix.

And three rules besides:

• The revenue guardrail. A winner whose revenue per visitor is clearly lower than the original’s (at least 10% lower, p below 0.05 on every visitor’s revenue, from at least 10 buyers between them) isn’t called. odo holds it (Held for you), says why, and leaves the call to you.
• Its days run out. When Call it after (28 days unless you change it, 7 to 90) passes without a call, it’s decided: no difference, with the plan’s numbers. One nobody was shown in all that time is stopped.
• Nobody shown it. A day after it starts with nobody shown it, odo says so, once: the page or the selectors probably don’t match.

A winner odo shows everyone stays a change odo makes: put it in your site’s code, then archive the experiment, and odo stops making it. Both kinds of winner wait among the Next moves for up to 30 days after the call: one odo shows everyone until you archive it, one in your code until you mark it shipped.

What you’re told#

Each call goes to your notification channels with Experiments on, under Tests: “B” won: +24% sign-ups, 97% sure. odo now shows “B” to everyone; make it permanent in your site’s code when you can. odo also tells you when it holds a winner, when an experiment is broken, and when nobody has been shown one.

Found in your code#

Experiments your site’s code ran in the last 90 days that nobody registered are listed under Found in your code (up to 10), with their visitors per variant. Adopt one and it opens in the form as an experiment in your code, with its name, page and variants: pick what it’s judged by and start it. odo reads it from its first exposure, so what it has run so far counts, and counts its days from when you adopted it.

What matched what#

Each experiment’s sheet breaks its result down by the channel, source, post or campaign (your links for posts and the posts odo found), device, new or returning visitor, and country of the visit each visitor was first shown it in: each version’s conversion there, and the leading version’s lift over the original.

Cut into enough pieces, any result shows differences by chance alone, so odo tests whether the leading version’s effect in a segment differs from its effect everywhere else, only where each version has at least 30 visitors in the segment and 30 outside it, with at least 5 converted on each side, and flags a segment only when that holds with q below 5%, allowing for every segment tested (Benjamini–Hochberg), and p below 1%. Even then it’s marked Worth testing: an idea for a test of its own, not a finding.

odo’s analyst#

On the Growth plan, and the free trial, odo’s analyst (Claude) writes a read-out when an experiment is called, and again when you ask (Read it out, at most every ten minutes): what happened and by how much, why it likely went that way, the segments it matched best and worst, the test to run next (Make the next test opens the form with it), and its caveats. It’s told the experiment’s definition and numbers, nothing else.

Suggest versions, in the form, proposes versions for the page and goal from the page’s last 30 days, what people click there and your past experiments, each change checked like yours, ready to start. On Starter you get the numbers alone, with a line saying so: odo never makes up a read-out. More in odo’s analysis with Claude.

Limits#

• 2 to 5 versions, the original first; up to 10 changes a version, each touching at most 20 elements.
• A website runs up to 20 experiments at once (running and paused together), and keeps up to 200 in all, archived ones included. Names stay taken, archived ones too.
• A hypothesis is up to 500 characters, a version’s name 60 and its description 300.

Reading the results#

The Experiments card on the Traffic dashboard lists the experiments odo runs (on the website on screen, when you pick one), newest first, each with its status, its versions and the call in a line. It reads each one since it started, whatever range is on screen. Tap one for its sheet:

• Result: the verdict in plain words, then for each version its visitors, how many converted and the rate with a 95% range, its lift over the control and, where it sells, revenue per visitor: “short” wins: +24% conversion, 97% sure. Ship “short” and remove “control”.
• Versions: what each one changes.
• What matched: the result by segment, as above.
• odo’s read-out, on Growth.
• In your site’s code: for an experiment in your code, the call and the markup to paste, with its name and keys.
• Running it: Show the winner to everyone, Call it after, and what can be done now: start, pause, resume, ship a version, stop or archive. Shipping a version the verdict hasn’t called a winner asks first.

How the verdict is called#

Reading a test every day and stopping when it looks good finds winners that aren’t. odo holds the call until the evidence is past a line that depends on how far along the test is.

The plan is how many visitors each variant needs to spot a 20% lift over the control’s rate, at 80% power. Before each variant has that many, only a difference past p below 0.001 is called; once they have, p below 0.05. Both lines are split over the variants compared with the control, so more variants don’t mean more false winners. The sheet says how far along the plan is, and about how many days are left at the current pace.

Until the verdict says winner, loser or no difference, don’t stop or ship: an early lead is often chance.

With an AI assistant#

An assistant connected to odo reads every experiment (list_experiments, analyze_experiment) and gets versions to try (suggest_experiment_variants). With Run experiments turned on when you approve it (it’s off unless you do), or a key with Experiments, it can also make and run them:

• create_experiment makes one, as a draft or started. In content mode odo changes the page; in code mode the answer has the exact call and markup for your code, under the same name and keys.
• update_experiment changes a draft, and starts, pauses, resumes, stops, ships or archives an experiment.

So a coding assistant can run the whole loop: find what to change, make it an experiment (or write it into your code and register it), check each version renders with ?odo_variant, and act only on odo’s call. odo tells it to ask you before starting, pausing, stopping or shipping one, since each changes what your visitors see. The improve_site and review_experiments prompts start it, and prompts to try has more.