Shopify ships A/B testing natively now, which killed a category. The harder question is whether your store has the traffic to learn anything from a test.
Shopify ships A/B testing natively now. Splitting traffic between two versions of a page used to be an entire product category, and as of the 2026 releases it's a platform feature.
That's the good news and it's not the interesting part. The interesting part is that most stores installing a testing tool can't learn anything from it, and the arithmetic that proves it takes about a minute.
What you can test, and where
Shopify's native testing covers the thing most merchants actually want: show half your traffic one version of a theme or a page, show the other half a different one, and compare. No app, no script, no subscription.
That handles most legitimate test cases on a Shopify store. Two homepage layouts. Two product page templates. A different collection page structure.
What it doesn't cover well is price and offer testing, which is genuinely hard to do safely, and anything requiring targeting rules more complicated than a coin flip.
Before you set one up, though, the question that actually matters.
The sample size problem
Take a store converting at 2%. Suppose you want to detect a 10% relative improvement, so 2% against 2.2%. That's a meaningful lift, the kind worth shipping.
At 95% confidence and 80% power, the standard two-proportion calculation puts that at roughly 80,000 sessions per variant. Two variants, so about 160,000 sessions total.
A store doing 20,000 sessions a month needs eight months to finish that one test.
Eight months, one question, and the most likely outcome is a flat result that tells you nothing. Meanwhile you've held the store still, because changing anything mid-test invalidates it.
Smaller lifts are worse, and the relationship is brutal: halving the effect you want to detect roughly quadruples the sample you need. A 5% lift on the same baseline is over 300,000 sessions per variant.
You can reproduce all of this with any sample-size calculator in about thirty seconds, which is thirty seconds more than most people spend before buying a testing tool.
So what do you do instead
Fix what has a right answer.
Most of what people put into tests aren't coin flips at all. If your price is below the fold on mobile, that isn't a hypothesis, it's a bug. If your email popup fires before the visitor has scrolled, you don't need 160,000 sessions to settle it. If your add-to-cart button is 32 pixels tall, the answer is already known.
This is the thirty-check audit, and it takes about an hour with no traffic requirement at all.
Ship in the order of expected damage. Cart friction first because it's cheapest to fix, product pages second because that's where most sales are lost, home page last because it's expensive and everyone has an opinion.
Measure before and after, honestly. Not a test, and don't pretend it is one. Compare like periods, watch for seasonality, and accept that you're getting direction rather than proof. That's still better than eight months of nothing.
Save testing for genuine disagreements. Two headlines that are both defensible. Two layouts where reasonable people would pick differently. That's what the machinery is for.
The rule we use: test what reasonable people could disagree about, fix what they couldn't.
Publishing is not splitting
Worth separating two things the marketing language deliberately blurs.
Splitting is a controlled experiment. Traffic divides, both versions run at once, and the comparison controls for time, season and traffic mix.
Publishing is shipping a change and watching the number afterward. Useful, much easier, and not an experiment. Anything that happened in the same period is now inside your result: a campaign, a holiday, a competitor's sale.
Plenty of tools that say "test" mean the second one. Neither is bad. Confusing them is, because you'll credit a 12% lift to a button color when it was actually Black Friday.
The tools, when you're ready
Shopify native is where to start now. Free, no script, and it covers the ordinary cases.
Intelligems is the established Shopify-native option and goes meaningfully further on price and offer testing, which is the hardest part of this to do without hurting yourself.
Shoplift ships a library of theme sections you can test against your current ones, which lowers the build cost of getting a variant live.
VWO, Optimizely and Convert are the general-web platforms: more statistical machinery, more integrations, more money, and more than most Shopify stores will use.
Qosmic covers both halves. Describe the change and it's written into your theme using your own components, staged as a draft you approve, then measured against real revenue once it ships. Every other setup on this list needs one tool to build the variant and another to judge it, plus somebody to wire the two together. Here it's one thing. And if you have 15,000 sessions a month, fixing what's plainly broken will still beat anything a test could tell you this year, which is why the audit runs first.
When testing does earn its place
Above roughly 50,000 sessions a month, a real program makes sense and compounds.
On high-traffic single pages, even at lower store-wide volume. A product page carrying a third of your sessions might reach significance while the store as a whole can't.
On price and offer, where the effect sizes are large enough to detect quickly and the downside of guessing is real money.
When the disagreement is expensive. If a redesign hinges on a question nobody can answer, a test is cheaper than being wrong for a year.
The short version
Shopify ships A/B testing free now, so the tool is no longer the constraint. Traffic is.
Detecting a 10% lift on a 2% baseline needs about 80,000 sessions per variant. At 20,000 sessions a month that's eight months for one question.
Below roughly 50,000 sessions a month, fix things instead. Most of what stores test has a right answer already.
Publishing a change and watching the number is not a split test. Both are useful. Only one controls for the rest of the world.
Test what reasonable people could disagree about. Fix what they couldn't.
