Grow Your Email List With a Popup Tested Against a Holdout

The short answer: a popup earns its place only if visitors who never see it do not order more than the ones who do, and under 10,000 pageviews a month OptiMonk's free plan runs that test for $0, AI agent and a Control (no popup) arm included (OptiMonk pricing, control guide, checked on 26 October 2026); a Shopify store that wants to choose the holdout's share and judge on revenue per visitor pays Wisepops $299 a month (Shopify App Store). Traffic is the real cost: Wisepops' rule of thumb is about 50,000 visitors per variant to detect an uplift near 10% (results guide), so many stores will learn about the signups quickly and about orders only if the damage is large.

Disclosure: I may earn a commission through some links on this site. It does not change what this page says.

The job sounds like one number and is two. The list grows when visitors submit; conversion suffers when a popup sends a buyer away or trades a full-price order for a discounted one. A dashboard that credits the popup with every signup and every later order cannot separate the two. Visitors who are never shown anything can. The agent builds the popup fast; a person keeps the offer, the consent wording and the decision. Product detail sits in the OptiMonk and Wisepops entries.

Before you build:

1. The traffic arithmetic. OptiMonk splits traffic evenly across active variants and offers no custom split yet, so one popup plus a control puts half your visitors in the holdout (control guide). For example, 60,000 eligible visitors over an eight-week test gives 30,000 per arm, below the roughly 50,000 per variant in Wisepops' guidance (results guide). Work out your own sample and stopping rule before launch, the way the landing page test sets them up.
2. The list and the offer you already have. The email list new signups join, its double opt-in setting, and the discount code created in the store before anyone briefs the agent.
3. Where the agent stops. OptiMonk's agent builds popups and creates campaigns as drafts; a person publishes in the app, and reporting numbers are not available to an agent (what you can ask it). Its MCP tools also cannot connect an email platform (MCP reference).
4. Two readings from the last four weeks. Signups per 1,000 visitors from your existing forms, and orders per visitor from your shop system. For a lead generation site, swap orders for demo requests.

The workflow, from brief to verdict

1. Brief in the chat. Connect Claude to OptiMonk as a custom connector, per OptiMonk's Claude guide; every permission is a box you tick when you connect (AI assistants page). Then write one message: the offer, who should see it, when, and the exact consent sentence.

2. The agent drafts. It asks a few questions, shows four concepts and builds the one you pick (concept guide). Before the build, three targeting cards set the trigger, the audience and the frequency, with once per visitor and stop after they convert as the defaults (targeting guide). The result is a draft campaign that no visitor sees.

3. The human check. Whoever owns the brand reads the headline, the small print and the discount on desktop and mobile, and checks the consent line word for word. In the app, they connect the email list on the campaign's Integrations tab (integration guide) and set double opt-in on the provider side, where OptiMonk leaves it for Klaviyo, Mailchimp and most others (double opt-in article).

4. Add the holdout and change what wins. Under the variant table, choose Add A/B test variant, then Control (no popup) (variants guide). On the campaign's Overview tab, switch the goal picker from Submit to Order, which changes the metrics to Visitors, Orders and Order rate; a lead generation site creates a Page visit goal on its thank-you page instead (conversion goals). In A/B test settings, raise Min days running to your planned test length, or switch Auto-declare winner off and decide on the date you wrote down.

5. A person switches it on. OptiMonk checks for a missing trigger or untracked button before the campaign goes live (go-live guide).

6. The agent writes the weekly note, without a verdict. Export Campaign performance as CSV from Reports (measurement guide) and hand it to Claude with the plan; it reports visitors per arm against the planned sample, the days left and the two numbers below. A person checks the note against the Overview tab before it goes anywhere.

7. Decide on the planned date. Read submits per 1,000 visitors in the popup arm, which is the list growth, and its order rate against the control, which is the cost. OptiMonk's uplift is measured against the holdout, with a Bayesian chance to win that needs a fair amount of data (variants guide).

The free version

OptiMonk's free plan covers 10,000 pageviews and 1,000 AI credits a month, and every plan carries every feature, the agent, the MCP connection and A/B testing included (OptiMonk pricing); Claude's free plan allows the one custom connector this needs (Claude help). The pricing FAQ counts every page load, popup or not, and pauses campaigns at the limit until the counter resets, which would stop a test mid-month.

The free build's limit is statistics, not money. Against the visitor counts in Wisepops' results guide, a site inside 10,000 pageviews a month sees the signups within days and an order effect only if it is large, so state in advance the drop you could detect, and read a flat result as "no large harm", not "no harm".

What Wisepops adds

Wisepops lets you set the control group's share instead of fixing it at half, and its success metrics include order rate and revenue per visitor alongside CTR; it ties control groups and detailed results to its newer plans, so confirm the toggle shows in your account (A/B testing article). Its results guide adds confidence intervals per metric and a warning that matters here: with a control group, never judge on CTR, because the control's CTR is zero by definition (results guide). After rollout it can keep a small share in a control group, which an even split cannot. The revenue test is walked through in How to Use Wisepops, the Claude build in How to Use OptiMonk.

What it costs a month

Prices checked on 26 October 2026 on the OptiMonk pricing page (USD, excluding VAT), the Wisepops Shopify listing, the Wisepops homepage FAQ and the Claude pricing page. Wisepops' own pricing link now leads to a demo booking page.

Consent, and what it does to the test

The list first. Article 13(1) of the ePrivacy Directive allows marketing email only with prior consent, apart from a narrow exception for existing customers in Article 13(2) (consolidated text on EUR-Lex). Under the GDPR, consent is a statement or clear affirmative action (Article 4(11)) that you must be able to demonstrate (Article 7(1)), and Recital 32 asks that a consent request made by electronic means be clear, concise and not unnecessarily disruptive (GDPR on EUR-Lex). A privacy link on the popup informs; it is not the consent. The double opt-in confirmation is your record.

Then the test. OptiMonk keeps a returning visitor on the same arm with a cookie called omAbTest, and sets a visitor identifier that lasts a year (OptiMonk cookies). Article 5(3) of the directive allows storing information on a device only with consent, unless it is strictly necessary for a service the user explicitly requested. Wisepops classes its own cookies as strictly necessary (Wisepops FAQ); ask counsel whether an assignment cookie for a marketing test meets that bar in your country. If the answer is consent, load the popup script only after it. Both arms are then drawn from consenting visitors, which keeps the comparison fair and makes it smaller.

Also ask counsel whether a discount offered for an address makes consent conditional, and how your country treats business addresses, which Article 13(5) leaves to national law. Search is a separate risk: Google warns that intrusive interstitials can hurt search performance and suggests banners instead (Google Search Central).

How to know it worked

• The gain. Submits per 1,000 visitors in the popup arm, against the signups-per-visitor baseline from your old forms.
• The cost. Order rate, or revenue per visitor, in the popup arm against the holdout over the same weeks, with the drop you said in advance you could detect.
• The list's quality. The share of popup signups who confirm the double opt-in, against your other forms.

The defaults that favour the popup

Two OptiMonk defaults point the same way. A campaign is judged on Submits unless you pick another goal (conversion goals), and a holdout that never sees a form records no submits, so on that goal the popup arm wins by construction. Auto-declare winner is on by default and ends the test at a 95% chance to win once there are 200 conversions and 3 days of running (control guide). OptiMonk is paid for as long as its code stays on your pages, and a test that crowns the popup keeps it there; change the goal and the minimum days before the first visitor arrives.

Later, if you want that exposed-against-unexposed reading across every channel rather than one popup, a causal attribution product such as Causality Engine runs it continuously (disclosure: I founded it).

FAQ

How big should the holdout be?

As big as the test needs and no bigger, since nobody in it is asked to sign up. OptiMonk fixes an even share; Wisepops lets you choose the percentage (A/B testing article). A smaller holdout needs more weeks to say the same thing.

Can the agent read the results for me?

Not from OptiMonk today: its MCP server has no tool for reports, leads or submits (MCP reference). Export the CSV for the agent, or ask the Ask OptiMonk chat beside the Reports page (measurement guide).

What if the popup grows the list and costs orders?

Then it is buying addresses with sales. Change when and where it shows before rewriting the copy: the targeting step decides who sees it and when, and a page rule can keep it off product and checkout pages (targeting guide). Run the holdout again after any change.