A/B testing: useful with traffic, misleading without it

How it works: two versions, random assignment, one measurement

“Test A/B” and “A/B test” are each searched 1,600 times a month in France. The principle: a version A (the current one) and a version B (the change), shown at random to comparable visitors over the same period. Since only the page differs, the measured gap in conversion can be attributed to the change, provided the sample is large enough.

Ronny Kohavi and his co-authors, who led experimentation at major web companies, sum up the value of the method in a landmark paper published in 2009: teams learn the most when they listen to their customers rather than to the “Highest Paid Person’s Opinion” (the “HiPPO”). The same paper stresses what makes tests fail: statistical power, sample size and variance.

How much traffic do you need to reach a conclusion?

The traffic you need depends on two things: your current conversion rate and the size of the gap you want to detect. The lower the rate and the smaller the gap, the more visitors you need. A common statistical approximation, for a two-version test with the usual thresholds (5% risk of a false alarm, 80% chance of detecting a real effect), gives per version: n ≈ 16 × p × (1 − p) ÷ d², where p is the conversion rate and d the absolute gap to detect.

A worked example: with a rate of 2% (p = 0.02) and the aim of detecting a rise to 2.4% (d = 0.004), you get about 16 × 0.02 × 0.98 ÷ 0.000016, so nearly 20,000 visitors per version, 40,000 in total, for a single test. A smaller gap needs far more.

Our own sites show the limit: across all our selling sites, Google Analytics measured 351 sessions from 1 October 2026 to 19 October 2026 (only visitors who accept measurement are counted). None of these sites can run a conclusive A/B test on purchases. The same goes for most small businesses.

Use a sample size calculator before you launch a test, and set the duration in advance. Stopping a test as soon as one version “wins” is the most common mistake: peek often enough and chance will always produce a false winner.

Without enough traffic: fix, then measure before and after

A small site gains more from fixing what is clearly broken than from testing nuances. Here is what you can fix without a test, because the flaw is obvious:

1. 1
   The buy or quote button that cannot be seen without scrolling on mobile.
2. 2
   Delivery costs that only appear at checkout.
3. 3
   A mandatory account before paying.
4. 4
   A form with more than five fields.
5. 5
   A page that takes several seconds to show its main content.
6. 6
   A phone number that cannot be tapped.

With traffic: what to test first

When your traffic allows it, test what can have the biggest effect, on the pages with the most visits:

• The offer itself: price, what a bundle contains, which plan is highlighted.
• The title and first sentence of the landing page.
• The order of the blocks: proof before or after the price.
• The number of checkout steps.
• How options are presented: three plans versus a single one.

Button colour rarely makes the list: the expected effect is too small to measure on a small business’s traffic.

Heatmaps and session recordings: useful, with consent

“Heatmap” is searched 5,400 times a month in France. A heatmap shows where visitors click and how far they scroll down a page. It proves nothing about conversion, but it quickly reveals problems: a non-clickable element that everyone tries to tap, a button nobody sees because it sits too low, a block that gets ignored.

On privacy, the CNIL (the French data protection authority) only exempts certain audience measurement tools from consent, under strict conditions (anonymous statistics only, data not reused for other purposes). Session recording and heatmap tools often go further: in France, check your tool’s configuration against the CNIL recommendation and collect consent when it is required.

Running a test properly: the checklist

1. 1
   A hypothesis written before the test: “showing the delivery time under the button will increase add-to-carts”.
2. 2
   A single primary metric, decided in advance: purchases, add-to-carts or quote requests.
3. 3
   Sample size and duration calculated before you start, in whole weeks to cover slow and busy days.
4. 4
   Random assignment of visitors, each one always seeing the same version.
5. 5
   No reading of the results to make a decision before the planned end.
6. 6
   A written report, winner or not: a test with no effect is information too.