Test a New Shopify Product Template Against a Holdout

The short answer: on Shopify's Grow plan, at $105 a month, the split costs nothing extra, because Rollouts runs a 50/50 experiment on a theme change in the admin; on Basic, assign the new template to a random half of your products instead, which costs $0 (Shopify Help Center, Shopify pricing). Pay for an app when the variant is built outside the theme editor or you want revenue per visitor as the goal: EComposer's Pro plan at $49 a month builds the template and tests it, and Shogun's A/B Testing Pro plan at $39 a month tests any product template you choose (monthly billing, checked on 26 October 2026 on the EComposer and Shogun pricing pages). Whichever splits the traffic, judge the result on orders per product-page visitor in Shopify's own order data.

Disclosure: I may earn a commission through some links on this site. It does not change what this page says.

One product template usually serves every product page in the store, so a new layout is a change to the whole catalogue at once. That is the reason to test it, and the reason to keep part of the store on the old one after it wins. Sample size and stopping rules are worked through in the landing page test, and the case for a no-change holdout in the popup test; this page covers what changes when the thing under test is a product template. The tools have their own entries: Shopify, EComposer and Shogun.

Before you build the variant:

1. The weeks your traffic buys. For example, if the calculator gives 7,400 visitors per arm and your product pages get 2,900 visitors a week, a two-arm test needs 7,400 x 2 / 2,900 = 5.1 weeks, so plan six whole weeks. If six weeks runs into your next sale, test a bolder change or wait.
2. The template you already run. The live template is the control and stays as it is. List the app blocks it carries, reviews, subscriptions, delivery dates, because a variant missing one of them tests the app, not the layout.
3. The measure and the guardrail. Orders per product-page visitor, or revenue per visitor, counted from Shopify's orders. The guardrail is returns on the orders placed during the test, read 30 days after it ends.

Three ways to split the traffic

Shopify Rollouts, on Grow and above. In Markets > Rollouts, create a rollout of the Experiment type, add a change with Edit current theme, customise the product template in the theme editor, then save and apply (create a rollout). A new experiment splits traffic 50/50 and ends 90 days after it starts unless you set another date (rollout types). Two limits matter here: a rollout cannot change Liquid templates, and it cannot be used on vintage themes (requirements).

A product-level split, on any plan. Create an alternate product template in the theme editor, then in the Products list use Bulk edit, add the Template column and assign the new template to a random half of your products (templates article). Nothing assigns visitors, so every order counts, consent banner or not. The cost is noise: products differ, so compare each group's change against its own last four weeks, not one group against the other.

A testing app. Shogun's Template Test compares two product templates, either a copy of the original or a template you select, and keeps each visitor on one variant for the whole test (first test article). EComposer's Template Test does the same for templates built in EComposer, with an adjustable split and a choice between showing a visitor one version for good or a fresh one each visit (EComposer A/B article). The full Shogun walkthrough is in How to use Shogun.

The workflow, with a builder and an agent

1. Draft the variant from product data. In EComposer, open Start Building > AI Product Page, choose Select from Store, pick a product and a template, and click Generate Page with AI. It maps the title, price, media and variants into the template, rewrites the product information in the template's tone, and has AI write stand-in copy for any section, such as an FAQ, that has no product data behind it (AI Product Page article).

2. The human check on the draft. The person who owns the product reads every line the product data did not supply. An AI-written FAQ answer about delivery, warranty or sizing is a claim your store now makes, so it is replaced with a true one or deleted. Check price and variants against the admin, the app blocks from your list, and the page on a phone.

3. The agent sets up the test, as a draft. On Shogun's Advanced testing plan, connect Claude to the Shogun MCP server at https://ai.shgapi.com/mcp and ask: "Create a template test on my product template, with my new template as the variant, Conversion Rate as the goal and a 50/50 split." The test is created as a draft that gets no traffic until you ask the assistant to start it and confirm, every write action is recorded in the test's audit log, and the guide recommends keeping write tools on Needs approval (Shogun MCP guide). A person opens the draft in the app and previews both variants before saying start.

   On EComposer instead, the test is set up by hand: connect Google Analytics 4, switch on the Shopify Pixel and embed the app in the theme, then create a campaign with the Template Test type, the split, the visitor behaviour and the targeting, and review the summary before it goes live (EComposer A/B article).

4. A weekly note, with no verdict in it. Each Monday the agent pulls the test's results and the orders attributed to each variant; Shogun's MCP can list the Shopify order ID of every attributed order, so the note can be checked against your own order export (Shogun MCP guide). The note reports visitors per arm against the plan and the weeks left. Nobody stops the test on a good Monday.

5. Decide on the planned date, then keep a holdout. A person reads orders per visitor for both arms, checks the returns guardrail when it comes in, and ships or drops the template. If it ships, leave a share of the store on the old one for four more weeks: a 90/10 split in Rollouts or EComposer, or a random tenth of products left on the old template. If the gap closes, the lift was novelty.

The free version

A store on Grow already has everything for the in-admin route: Rollouts, the theme editor for the variant, and Shopify's order data for the verdict. A store on Basic has the product-level split, the bulk editor and the same order data. Either is enough on its own for a layout change made in the theme editor. Rollouts reports conversion rate over time, bounce rate, reached checkout rate and add to cart rate for a theme experiment, with no revenue figures; it counts only visitors who consent, and drops anyone shown both versions in the same browser (rollout analytics). In the EU, that consenting share is your sample.

What the apps add: a template built in a drag-and-drop builder rather than the theme editor, revenue per visitor and order value as goals, and an agent that can set up and report on tests.

What it costs a month

Prices in USD, monthly billing unless noted, checked on 26 October 2026 on the Shopify, EComposer and Shogun pricing pages; the connector limit is from the Shogun MCP guide.

Where the free test sits

Shopify keeps experiments for the Grow plan and above, so the test built into the admin costs a Basic store $105 - $39 = $66 a month in plan difference before any app is installed (requirements, Shopify pricing). Upgrading for one test is a plan decision dressed as a testing decision; the product-level split answers the same question on Basic.

How to know it worked

• The measure. Orders per product-page visitor in each arm, from Shopify's orders. Shogun's Conversion Rate goal tracks visitors who buy after landing on a tested page, while its Add to Cart Rate counts additions anywhere in the session (goal definitions); a layout can raise additions and leave orders where they were.
• The baseline. Four weeks of orders per product-page visitor before launch, for the whole store and, on the product-level split, for each half.
• The holdout. The control arm during the test, and the 10% left on the old template for four weeks after.
• The match. Each tool counts a different crowd: Rollouts only visitors who consent (rollout analytics), Shogun only shoppers who take part in the experiment (Shogun analytics article), EComposer what Google Analytics 4 and the Shopify Pixel pass it (EComposer A/B article). Match the order IDs a tool credits to each arm against your export before believing its lift.

Later, if you want that match between what a tool reports and what the business booked run continuously across every channel, a causal attribution tool like Causality Engine does it (disclosure: I founded it).

FAQ

Can Shopify Rollouts test a template made in a page builder?

Shopify's requirements say a rollout cannot change Liquid templates (requirements). Ask the builder's support whether its template is something a rollout can carry, or test it in the builder's own Template Test, or in Shogun's split URL test, which compares two existing pages, including a new version built in another app (test types).

Is add to cart rate good enough as the goal?

Only as an early signal. More shoppers add to cart than buy, so it reaches a verdict sooner, and Shogun counts an addition anywhere in the session (goal definitions). Ship on orders per visitor.

How long should the holdout run after launch?

Four weeks is enough to see whether the gap holds once the novelty wears off, and short enough that few shoppers see the old page. Applying a rollout's changes cannot be undone (managing rollouts), so keep the old template in the theme until the holdout ends.