Open In A New Tab

Ron Kohavi, Lukas Vermeer and Jakub Linowski predict that opening product pages in a new tab helps shoppers compare. Yet no high-powered experiment has confirmed it: attempts at Metro and Obs only showed directionally positive signals. Here’s what 8 past experiments found so far:

High power estimate +2.2%
Highest estimate +6.4%
Lessons 6
88% positive · 8 tests
Taught
by
Ron
Kohavi
Jakub
Linowski
Recent replications
Andrey · Metro Test #666 +0.3% adds to cart Stat-sig Pre-reg+2.6% sales Stat-sig Pre-reg
Joachim · Obsbygg.no Test #644 -1.5% adds to cart Stat-sig Pre-reg
Joachim · Obs.no Test #645 +2.6% adds to cart Stat-sig Pre-reg

We think this pattern can, and should, replicate.

Same tab: Bol.com & Amazon

Bol.com and Amazon search results, which open product pages in the same tab

New tab: Etsy & Airbnb

Etsy and Airbnb search results, which open listing pages in a new tab

Start From an Instructor’s Setup

Each instructor sets the test up their way. Pick one, then adjust anything below.

A frequentist, high-powered replication that stands on its own: 90% power, two-sided, fixed horizon, no pooling.

Meta-power is fine, since every bit helps: run it, share it, and add more behavioral metrics to see what users are really doing.

Targeting / Triggering

Refine where and when the test fires, beyond the page types chosen above in order to increase power (and shorten the testing duration).

Metrics & Ship Requirements

Mark each metric as Primary, Secondary, or Guardrail, then set the requirement it must meet to ship.

Metric Primary Secondary Guardrail Superiority Non-Inferiority Binary Continuous
%
%
%
%
%
%
%

Power, Sample & Duration

The effect you want to detect and how long you'll run to reach the power for it.

%
%
%
%
Total Sample — visitors across all variants
Duration —weeks  

Meta-Analysis

Combine your experiment with that of others to reach a decision faster or increase power.

Start Rules

When can the experiment start?

Stop Criteria

When to end the experiment. Pick both for a hybrid.

Stop at

Segments

Which subgroups should we track and break the results down by, to see if the effect differs across them?

Segment Exclude Measure
Desktop: May be more effective here / Mobile: May be less effective — switching tabs is more difficult
Listing page: May be more effective on product tiles linking to detail pages / Search page: May be more effective on product tiles linking to detail pages / Category reference: May be less effective when linking to 2nd-level category pages
101+: May be more effective with many more results to compare
First-time buyer: May be more effective here
Yes: May be less effective — the detail page already enables comparison / No: May be more effective
0: May be more effective — comparison isn't already enabled
High: May be more effective — comparison across tabs matters more when products look similar

Iteration Rules

What next experiments to run once the results come in.

Outcome → Re-run with: Higher intensity Higher power Elsewhere Something different Invert Move on
IF Stat sig primary A clear, significant win.
IF Stat sig primary Flat secondary A win on primary, no effect on secondary.
IF Flat primary Stat sig secondary No effect on primary, a win on secondary.
IF Flat primary Flat secondary No significant effect on either.
IF Stat sig neg primary A clear, significant loss.

Export to Experimentation Tools

Pick a destination — copy a plain-text spec, or a JSON config for your platform.


        
        
    

Now Run This Experiment