Open In A New Tab
Ron Kohavi, Lukas Vermeer and Jakub Linowski predict that opening product pages in a new tab helps shoppers compare. Yet no high-powered experiment has confirmed it: attempts at Metro and Obs only showed directionally positive signals. Here’s what 8 past experiments found so far:
We think this pattern can, and should, replicate.
Same tab: Bol.com & Amazon
New tab: Etsy & Airbnb
Start From an Instructor’s Setup
Each instructor sets the test up their way. Pick one, then adjust anything below.
A frequentist, high-powered replication that stands on its own: 90% power, two-sided, fixed horizon, no pooling.
Meta-power is fine, since every bit helps: run it, share it, and add more behavioral metrics to see what users are really doing.
Targeting / Triggering
Refine where and when the test fires, beyond the page types chosen above in order to increase power (and shorten the testing duration).
Metrics & Ship Requirements
Mark each metric as Primary, Secondary, or Guardrail, then set the requirement it must meet to ship.
| Metric | Primary | Secondary | Guardrail | Superiority | Non-Inferiority | Binary | Continuous |
|---|---|---|---|---|---|---|---|
| % | |||||||
| % | |||||||
| % | |||||||
| % | |||||||
| % | |||||||
| % | |||||||
| % | |||||||
| % |
Power, Sample & Duration
The effect you want to detect and how long you'll run to reach the power for it.
Enter a baseline rate and MDE to estimate the sample size.
Meta-Analysis
Combine your experiment with that of others to reach a decision faster or increase power.
Start Rules
When can the experiment start?
Stop Criteria
When to end the experiment. Pick both for a hybrid.
Segments
Which subgroups should we track and break the results down by, to see if the effect differs across them?
| Segment | Exclude | Measure |
|---|---|---|
|
Desktop: May be more effective here / Mobile: May be less effective — switching tabs is more difficult
|
||
|
Listing page: May be more effective on product tiles linking to detail pages / Search page: May be more effective on product tiles linking to detail pages / Category reference: May be less effective when linking to 2nd-level category pages
|
||
|
101+: May be more effective with many more results to compare
|
||
|
First-time buyer: May be more effective here
|
||
|
Yes: May be less effective — the detail page already enables comparison / No: May be more effective
|
||
|
0: May be more effective — comparison isn't already enabled
|
||
|
High: May be more effective — comparison across tabs matters more when products look similar
|
Iteration Rules
What next experiments to run once the results come in.
| Outcome | → Re-run with: Higher intensity | Higher power | Elsewhere | Something different | Invert | Move on |
|---|---|---|---|---|---|---|
| IF Stat sig primary A clear, significant win. | ||||||
| IF Stat sig primary Flat secondary A win on primary, no effect on secondary. | ||||||
| IF Flat primary Stat sig secondary No effect on primary, a win on secondary. | ||||||
| IF Flat primary Flat secondary No significant effect on either. | ||||||
| IF Stat sig neg primary A clear, significant loss. |
Export to Experimentation Tools
Pick a destination — copy a plain-text spec, or a JSON config for your platform.
Now Run This Experiment
- Unlock the evidence behind each variable
- All evidence-backed defaults revealed
- Design review
- Power and metrics review
- Targeting and scope review
- Full implementation and QA
- Test run and monitoring
- Result review and write-up
A GoodUI subscription unlocks the A/B test evidence behind every variable in this design, plus all of its evidence-backed defaults, so each choice is backed by data from similar experiments.
Beyond the review, we teach you or your team the reasoning behind it, so the next test is one you can design and judge yourselves. Pick whichever lessons you're interested in:
Hand it off. We design, implement, and QA the test, run and monitor it for you, then review the results and deliver a write-up — you get the outcome without the build effort.
- Metric alignment and setup Define the primary metric, guardrails, target audience, tracking events, attribution window and launch criteria so the test measures the decision you actually need to make.
- Power calc Estimate baseline conversion, minimum detectable effect, sample size and runtime before launch, with a clear read on whether the test is likely to be decisive or exploratory.
- Prototype-quality code Build the variation to prototype quality in your testing tool or front-end stack, with responsive states, analytics hooks, QA notes and handoff-friendly implementation details.
- Analysis Read the result with significance, confidence intervals, guardrail checks and segment caveats, then translate the stats into a practical ship, stop or iterate recommendation.
- Post-test and iteration recommendations Turn the result into next moves: what to keep, what to change, what to retest and which follow-up experiments are most worth running next.
- Discussions Use async or live discussion time to clarify assumptions before launch, review tradeoffs during the run and unpack ambiguous outcomes after the test closes.
- Design
- Power & MDE
- Metrics & guardrails
- Targeting & scope
- Iteration suggestions