A/B Test

Open In A New Tab

Ron Kohavi, Lukas Vermeer and Jakub Linowski predict that opening product pages in a new tab helps shoppers compare. Yet no high-powered experiment has confirmed it: attempts at Metro and Obs only showed directionally positive signals. Here’s what 8 past experiments found so far:

QuestionDoes opening links in a new tab allow more product comparisons, which in turn increases purchase rate and basket size?

High power estimate +2.2%
Highest estimate +6.4%
88% positive · 8 tests
With
defaults
from
Ron Kohavi Ex-Airbnb, Microsoft & Amazon Jakub Linowski Founder of GoodUI

Run this experiment for: 6 lessons ImpactNoiseVelocityExplorationIterationQuality

Recent replications
Andrey · Metro Test #666 adds to cart Stat-sig
Joachim · Obsbygg.no Test #644 adds to cart Stat-sig
Joachim · Obs.no Test #645 adds to cart Stat-sig

Bol.com & Amazon

are opening product/listing links in the same window

Bol.com and Amazon search results, which open product pages in the same tab

Etsy & Airbnb

are opening product/listing links in a new tab

Etsy and Airbnb search results, which open listing pages in a new tab

Experiment Design

Choose the structure of your experiment.

Treatment Variables

Tick what each variant changes, or save it for a follow-up experiment. Some choices are backed by past tests; others are still hunches. Impact

Treatment variableInclude in B C D E Future
Evidence ↑ 2 of 3 tests (Tests 666, 644, 645)
the browser default
Exploratory sets the expectation
Exploratory shows what's already open
Evidence ↑ 1 of 2 tests (Pattern #92)
Exploratory keeps the comparison going
Exploratory run as its own variant

Targeting / Triggering

Refine where and when the test fires, beyond the page types chosen above in order to increase power (and shorten the testing duration). NoiseVelocity

Variance Reduction

Cancel out noise you can predict, to get more power from the same traffic. NoiseVelocity

Helps returning visitors with history; new visitors add little.

th percentile

Set the cap before the test starts: capping slightly changes what is measured.

Metrics & Ship Requirements

Mark each metric as Primary, Secondary, or Guardrail, then set the requirement it must meet to ship. ImpactIteration

Metric Primary Secondary Guardrail Superiority Non-Inferiority Binary Continuous
%
%
%
%
%
%
%
%
%

Power, Sample & Duration

The effect you want to detect and how long you'll run to reach the power for it. Noise

%
%
%
%

Total Sample — visitors across all variants
Duration —weeks  

Meta-Analysis

Combine your experiment with that of others to reach a decision faster or increase power. NoiseVelocity

Start Rules

When can the experiment start?

Stop Criteria

When to end the experiment. Each early stop trades some power for a shorter run. Quality

Stop at

Segments

Which subgroups should we track and break the results down by, to see if the effect differs across them? Iteration

Segment Your Bet Exclude Measure
Likely stronger
Likely weaker: tab switching is harder
Likely stronger: tiles link to products
Likely stronger: tiles link to products
Likely weaker: links go to categories
Likely stronger: more to compare
Likely stronger
Likely weaker: the page already compares
Likely stronger
Likely stronger: nothing to compare with yet
Likely stronger: similar products need comparing

Iteration Rules

What next experiments to run once the results come in.

Outcome Re-run with: Higher intensity Higher power Elsewhere Something different Invert Move on
IF Stat sig primary A clear, significant win.
IF Stat sig primary Flat secondary A win on primary, no effect on secondary.
IF Flat primary Stat sig secondary No effect on primary, a win on secondary.
IF Flat primary Flat secondary No significant effect on either.
IF Stat sig neg primary A clear, significant loss.

Export to Experimentation Tools

Pick a destination — copy a plain-text spec, or a JSON config for your platform.


            
            
        

Now Run This Experiment