How to Use Optimizely to Test Three Checkout Variations

The short answer: Stats Accelerator needs at least two variations plus the baseline, so a checkout test with one challenger runs on manual allocation and a test with two can hand the split to the accelerator, which Optimizely says reaches significance two to three times faster than a fixed split and reports improvement in percentage points to correct for the moving allocation (Stats Accelerator overview). Install the snippet sitewide at the top of the head, create the A/B test on the checkout URL, build two variations in the Visual Editor, add the order event as the first metric, set Distribution Mode to Stats Accelerator, preview, start, read the results page when the confidence interval clears zero, and have Mark write the summary over the MCP server while a person checks the Experiment Health line before anything is paused.

Disclosure: I may earn a commission through some links on this site. It does not change what this page says.

Optimizely publishes no price, every plan is individually packaged (Optimizely plans, checked on 5 November 2026), so the steps below assume an account with Web Experimentation; the Optimizely glossary entry sorts out which product is which. The job: two ideas for the payment step, one original, and a decision in weeks rather than a quarter.

Before you start:

1. Count what the third version costs in traffic. For example, a checkout step with 30,000 visitors a week split three ways under manual allocation gives 10,000 each; under the accelerator the counts drift every hour toward the variation furthest from the baseline, and the figure you report is absolute improvement in points (Stats Accelerator overview). If you have one idea, do not invent a second to qualify.
2. Start from the event you already fire. The first metric you add becomes the primary metric and decides the winner, so make it the order-complete event, tracked before the experiment starts (steps to create an experiment).
3. Write down what done means. A confidence interval entirely above or below zero on the primary metric, Experiment Health showing Good, and results that arrive five to ten minutes after ingestion (results page).

What you need

• A Web Experimentation account and a role that can publish: project owners publish and pause, editors build (get started).
• Access to the head of every page, or the template that renders it.
• An order-complete event or a thank-you page to measure.
• For the agent step, Mark (Optimizely Agent Platform) enabled on the account and an AI client that supports remote MCP (MCP quickstart).

1. Install the snippet once, sitewide

Under Settings, Implementation, copy the snippet and paste it at the top of the head on every page; the docs say to add it sitewide and not to change a character of it (get started). Each project has its own snippet.

2. Create the test on the checkout page

In the Experiments dashboard, click Create New Experiment, choose A/B Test, and give it a name. Target By URL runs the test on one address; Target By Saved Pages runs it on a set of URLs you will reuse (steps to create an experiment). A checkout step with a stable URL takes the first.

3. Build two variations

Under Design, Variations, add two variations and open each in the Visual Editor to change layout, styling or text; upload a screenshot for each so reviewers see what changed (steps to create an experiment). With the original that makes three, the accelerator's minimum.

4. Leave the audience at Everyone

The default audience is Everyone; add saved audiences with AND or OR only if the test should exclude someone, say logged-in staff. Do not add an audience to read results by device later: Web Experimentation does not segment results by audience, and device comes from the results page's own segments (segment your results).

5. Add the order event first

Under Track, Metrics, click Add Metric and choose the order-complete event. The first metric is the primary metric; secondary metrics and monitor goals are optional, and at least one metric is required before the experiment can run (steps to create an experiment).

6. Switch the distribution mode

Under Design, Traffic Allocation, set Distribution Mode to Stats Accelerator. The model behind it updates hourly, looks for the variation most distinct from the baseline, positive or negative, and drops a variation from the pool once it reaches significance (Stats Accelerator overview). Three rules from the same page: set the primary metric before starting; never run an A/A test on it; and do not switch to manual allocation mid-experiment, which the docs say introduces severe bias.

7. Preview, review, start

Click the Preview icon to check each variation renders; if nothing shows, the snippet is missing. Set start and end times under Plan if you want them, run the pre-launch review with Mark, then click Start Experiment (steps to create an experiment).

8. Read the results page

Per variation you get visitors, conversions, the conversion rate, the improvement against the baseline and a confidence interval that narrows as data arrives and is conclusive once it sits wholly above or below zero (results page). On an accelerator experiment the improvement is absolute, in points, with an approximate weighted improvement under it for the winner. Experiment Health checks the traffic split; anything other than Good means stop reading and investigate. Segment by device or source for exploration only, and leave Reset Results alone: it is permanent.

9. Let Mark summarise, let a person decide

Connect the Experimentation MCP server, in Claude Code with claude mcp add --transport http optimizely-exp https://exp.mcp.opal.optimizely.com/mcp, or through the MCP settings of Claude Desktop, Claude.ai or Cursor; the first call opens an OAuth window where you sign in with your Opti ID and pick the Mark instance (MCP quickstart). Start with "List my Optimizely projects", then ask for a summary of the checkout experiment: the query tools include exp_summarize_test_result for lift and significance (MCP overview).

The human check has two parts. The Agent Platform's own tools page says Mark cannot look up detailed results for a specific experiment by name and that results support is still evolving, so the summary is a draft and the results page is the record (system tools). And the server can create and update experiments, flags and audiences, confirming the project and environment before any write, while variation code changes only in the Visual Editor. Nobody pauses a variation on the strength of a chat message.

10. File the hypothesis and the outcome

Collaboration holds the hypothesis with its brief, variations and results, and takes ideas through a submission form (Collaboration docs). Reporting for Experimentation gives a Program Overview of experiments by status and a Metric Impact report with a win rate, for any Opti ID user with a role on the instance (reporting dashboards).

What the docs warn about

• Pausing a variation mid-test skews weighted improvement through delayed conversions; let the accelerator starve a loser instead (Stats Accelerator overview).
• Changing the primary metric mid-test changes the policy, since the accelerator is tuned to it.
• Republished snippets. The accelerator republishes the snippet periodically, and that publishes any pending changes in the project with it, so keep drafts out of a live project.
• Accelerated tests vanish from the programme view. Top experiments, underperforming experiments and win rate do not include experiments run with Stats Accelerator as the distribution mode (system tools).

What speed is for

The accelerator's product is time, and time is also what Optimizely gains from it: a test that concludes in a third of the weeks hands the checkout to the next test sooner, which is how a platform arrives at "2m+" experiments run (experimentation page). The slow part is left with you: accelerated tests drop out of the win-rate and top-experiment reports Mark draws from, so keep a log of them with the hypothesis, the points gained and the dates, or the fastest tests become the ones nobody can find.

FAQ

Do I need three variations?

Yes, for Stats Accelerator: two variations plus the baseline, in Web Experimentation and in Feature Experimentation alike (Stats Accelerator overview). With one challenger, use manual allocation.

Why is the improvement shown in points?

Because reallocating traffic over time biases a relative figure, a form of Simpson's paradox; the docs report absolute improvement and compute a weighted improvement per epoch of constant allocation (Stats Accelerator overview).

Can Mark change a running experiment?

It can update experiments, flags and audiences after confirming the project and environment with you, and it cannot touch variation code, which stays in the Visual Editor (MCP overview).