Smoke Testing Product Ideas: A Method for Many Variants

The short answer: a smoke test shows real buyers a page for a product that does not exist yet and counts what they do. To test many variants at once, describe them as a matrix, generate one page per variant from a template, give each the same small ad budget, and rank the variants by the most committing action you can honestly offer. With many variants and small budgets the result is a ranking, not proof, so test in stages, agree the pass line first, and tell visitors the truth: the product is not available yet.

At Procter & Gamble I smoke-tested 128 product variations in 45 days to launch a new product line. This post describes the method as I would run it. It is not an account of that test, and the worked example below is invented and labeled as such. For where a smoke test sits in a wider plan, see corporate venture building.

Why measure what people do

What people say they would buy and what they buy are different things. Vicki Morwitz's review of 60 years of research concludes that purchase intentions "are correlated and predict future sales, but do so imperfectly". In a meta-analysis of 28 studies that measured willingness to pay both hypothetically and for real, the median hypothetical value was 1.35 times the actual one, and the distribution was heavily skewed, so some studies found far larger gaps.

A smoke test swaps the question for an action. An action is still not a sale, and the method below is built around that gap.

When a smoke test is the wrong tool

• The value shows only in use, so a page cannot carry it. Build a prototype or deliver the service by hand first.
• You cannot reach enough buyers with paid traffic. For a few hundred named accounts, interview them.
• The product is regulated, such as financial or health products, and advertising it before launch may be restricted. Ask legal first.
• The decision is already made. A test whose result will be ignored is theater.

The method in six steps

1. Write the decision and the pass line first

Decide what the test will change: which variants you build, which you drop, and what happens if none passes. Then write the pass line before any traffic arrives. Strategyzer's Test Card has four fields for it: the hypothesis, the test, the measurement and the success criterion, which it frames as a threshold.

Derive the threshold from the economics:

required sign-up rate = cost per click ÷ (affordable cost per customer × share of sign-ups who become customers)

Illustration, not a client case: you can afford €80 to win a customer, you assume one in five sign-ups buys, and a click costs €0.80. A sign-up is then worth at most €16 (80 × 0.2), so at least 5 percent of clicks must turn into sign-ups (0.80 ÷ 16). If the required rate is far above what your own paid pages usually achieve, change the price or the economics, not the ad. The break-even ROAS calculator shows the most you can pay per order, and the LTV:CAC calculator checks whether a cost pays back. The share of sign-ups who buy stays an assumption until stage 3.

2. Turn the idea into a matrix

List what could change the buying decision and what a page can show: the promise, the price, the design or format, the audience. Give each axis three or four levels, and keep only levels you would be willing to build. The number of variants is the product of the levels.

3. Generate the pages from one template

One template, one script or page builder, one page per variant. Every page shows a working name, the promise, the expected price, an image that is clearly a concept, one action, and a plain statement that the product is not on sale yet (the section on the honest page below says how). Keep the layout identical, so that differences in results come from the variable and not the design. Tag every ad and page with a variant ID so you can slice results by promise, price and design later.

4. Give every variant the same small budget

Same audience, same placements, same dates, same budget per variant. Set the budgets yourself. Meta describes its automatic campaign budget as one budget distributed across ad sets in real time to find the best opportunities, which moves spend toward early leaders and ends the equal comparison. Exclude staff, agency and test traffic.

5. Read the signals on a commitment ladder

The more a signal costs the visitor, the less it overstates demand. An email address costs almost nothing, so I treat a sign-up as an upper bound on interest and use it to rank variants, not to forecast sales. The ordering is my judgment, resting on the two studies above: stated intent is an imperfect guide to purchases, and hypothetical values run above real ones.

6. Stage the test and read it by attribute first

Do not pick winners from a noisy ranking of 128 pages. Stage 1 pools every page that shares a promise, a price or a design and reads the matrix by attribute. Stage 2 retests the best levels as a short list, with bigger budgets. Stage 3 puts the finalists through the most committing test you can honestly run, a refundable deposit or pre-order, against the pass line from step 1. On a smaller budget, run a balanced fraction of the matrix. NIST's handbook shows the principle: main effects can be estimated from half the runs of a three-factor design, provided interactions are small next to them.

Illustration, not the P&G test: 128 variants of an invented product

Everything in this section is invented to show the arithmetic: the product, the click cost and the conversion rate. None of it is a result from a real test.

Setup. An invented reusable water bottle. Four promises (cold for 24 hours, dishwasher safe, recycled steel, leak-proof in a bag), four expected prices (€19, €24, €29, €34) and eight designs give 4 × 4 × 8 = 128 variants, and 128 pages built from one template. The budget is €100 per variant, €12,800 in total. At an assumed €0.80 per click, each page gets 125 clicks. Assume every variant truly turns 5 percent of clicks into sign-ups, so the expected count is 6.25 sign-ups per page.

What one variant tells you. Six sign-ups from 125 clicks is 4.8 percent, and its 95 percent confidence interval (Wilson method) runs from 2.2 to 10.1 percent. A page needs about 12 sign-ups (9.6 percent, interval 5.6 to 16.0 percent) before its interval clears a 5 percent baseline. A single page has to convert at nearly double the baseline to stand out.

What a ranking looks like when nothing differs. Suppose all 128 variants are identical, each with a true rate of 5 percent. Chance alone gives about three of them 12 or more sign-ups: the probability for one page is 2.3 percent, and 128 × 0.023 is about 3. Across repeated runs the best page has a median of 13 sign-ups (10.4 percent), and in 95 runs out of 100 it reaches at least 12. A top three picked from that ranking is noise, even though a page with 12 sign-ups clears the baseline on its own interval. The interval does not allow for the fact that you picked the page for being the highest of 128.

What the same money can read. Pool by attribute. Each promise appears on 32 of the 128 pages, so it collects 32 × 125 = 4,000 clicks and about 200 sign-ups, with an interval of plus or minus 0.7 points around 5 percent. Each design appears on 16 pages: 2,000 clicks, plus or minus 1.0 point. A single page, by contrast, is known only to within a range of about 2 to 10 percent. So the matrix can rank promises, prices and designs to within about a point, but it cannot rank single pages. This read assumes the attributes add up, with no strong interaction such as a promise that works only at one price. Stage 2 checks that.

The stages.

For comparison, reading every one of the 128 pages to within plus or minus 2 points at 5 percent needs about 460 clicks each (1.96² × 0.05 × 0.95 ÷ 0.02² = 456), roughly €365 per page and about €46,700 in all. The staged route reaches a short list for €18,800 (12,800 + 6,000).

Reading the result

Treat the output as ranking evidence, not proof.

• A variant that clears the pass line in stage 1 is a candidate for stage 2, not a winner, and finalists tend to fall back when retested.
• A null result is a result. If nothing clears the pass line in stage 2 or 3, stop or change the proposition. Do not rerun the same test until something passes.
• The test measures the response of people your ads reached. It ranks variants. It does not size the market or forecast sales.

Do not tell visitors the product is sold out

A page that shows a price and a purchase button for a product you cannot supply is a false offer, and a "sold out" message adds a false statement about availability. Two sets of rules apply.

EU consumer law. The Unfair Commercial Practices Directive treats a practice as misleading if it contains false information, or deceives the average consumer, about a product's main characteristics, "such as its availability", and so leads to a decision the consumer would not otherwise have taken (Article 6(1)). A page that states characteristics and a price and lets people buy is an "invitation to purchase" (Article 2(i)). Among the practices the Directive treats as unfair in all circumstances is inviting people to buy at a specified price without disclosing that you have reasonable grounds to believe you cannot supply the product at that price, for a reasonable period and in reasonable quantities (Annex I, point 5, "bait advertising"). This is not legal advice, and enforcement differs by country, so take advice before you test with consumers.

Ad platforms. Google Ads' Misrepresentation policy has a section on unavailable offers: promising products, services or offers in an ad that are unavailable is not allowed. Meta's Advertising Standards prohibit ads that promote products or offers using deceptive or misleading practices. Policies change, so read the current text before you launch. A rejected ad ends the test before it starts.

The honest page. Build it like this:

1. A line above the form: "This product is not on sale yet."
2. The price you expect to charge, labeled as expected. Give a launch window only if you would meet it; otherwise say you are deciding whether to make it.
3. A button that says what it does: "Join the waitlist".
4. A form that asks for an email address and, for a stronger signal, one choice: a pack size, a price band or the intended use.
5. A privacy notice where the address is collected. GDPR Article 13 lists what to tell people at that moment: your identity and contact details, the purposes and legal basis, who receives the data, how long you keep it, and their rights, including to withdraw consent and to complain to the regulator. Use the address only for what you said (Article 5(1)(b)) and ask for no more than you need (Article 5(1)(c)). Example: "Your address is used to tell you once when this is available and to ask up to three questions. It is deleted after six months if the product does not launch."
6. An ad that says the same as the page: "Join the waitlist" or "Coming soon".

A paid pre-order is a sale, and its page still says plainly that the product is not yet available. For an invitation to purchase, the Directive treats as material the main characteristics, the trader's identity and geographical address, the price including taxes and delivery charges, and any right of withdrawal (Article 7(4)). State the delivery window and the refund terms, be sure you can refund, and take legal advice first. That is why the deposit test belongs in stage 3, not on all 128 pages.

Why the honest page is also the better test

• The sign-up means more. The visitor knows the product is not available and joins anyway, so you know what they knew. A "sold out" message adds a popularity cue that has nothing to do with your product, and you could not separate the two effects.
• You can follow up truthfully. A real waitlist can be asked a price question or offered a refundable deposit, which is stage 3. People told the product sold out cannot be asked to pay for it.
• Nothing to clean up. An ad that says "join the waitlist" makes no claim of availability, which is what Google's policy targets, and you do not end up holding a list of people you told something untrue.

Frequently Asked Questions

What is the difference between a smoke test and a focus group or survey?

A smoke test counts what people do, while a focus group or survey records what they say. Stated intent is a useful but imperfect guide: purchase intentions predict sales imperfectly, and in the willingness-to-pay meta-analysis the median hypothetical value was 1.35 times the actual one. A smoke test is not a sale either, which is why the method climbs a ladder of commitment.

Is it legal and ethical to test a product that does not exist?

It can be, if the page tells the truth. Say the product is not available yet, do not claim it is sold out or scarce, show the price as expected, collect only what you need and say what you will do with it. A page that invites people to buy something you cannot supply risks breaching EU consumer law and ad-platform policies, as set out above. This is not legal advice. For B2B tests different rules apply, and the honest design is still the right one.

How much does a smoke test cost, and how many variants can I test?

The cost is clicks times cost per click times variants, plus the time to build the pages. At an assumed €0.80 per click (use your own), reading one variant to within about 2 points at a 5 percent sign-up rate takes about 460 clicks, around €365. A budget of €12,800 therefore reads about 35 variants, not 128. Twelve variants cost about €4,400 and 128 about €46,700, which is why I stage the test and read a big matrix by attribute first.

Can this method be used for B2B products or services?

Yes, with changes. The action becomes "join the early-access list" or "book a call". Audiences are smaller, so test fewer variants, promise and price first, and add interviews with named accounts. The honest-page rules are the same.