A/B Test

Run Your Best Sticky Call To Action Experiment

With 194,400 possible variable combinations across 29 experiments, past lifts have ranged:

High power estimate +1.5%
Highest estimate +9.1%
72% positive · 29 tests
With
defaults
from
Jakub Linowski Founder of GoodUI

Scope

Choose the page type where the test runs.

Prototype

PHOTO

Specs

Reviews

Treatment Variables

Set each option, then check the variables your B version changes.

•
•
•
1 test

Experiment Design

Choose the structure of your experiment.

Targeting / Triggering

Refine where and when the test fires, beyond the page types chosen above in order to increase power (and shorten the testing duration).

Variance Reduction

Cancel out noise you can predict, to get more power from the same traffic.

Metrics & Ship Requirements

Mark each metric as Primary, Secondary, or Guardrail, then set the requirement it must meet to ship.

Metric Primary Secondary Guardrail Superiority Non-Inferiority Binary Continuous
%
%
%
%
%
%
%
%
%
%
%
%
%
%

Power, Sample & Duration

The effect you want to detect and how long you'll run to reach the power for it.

%
%
%
%

Total Sample — visitors across all variants
Duration —weeks  

Meta-Analysis

Combine your experiment with that of others to reach a decision faster or increase power.

Start Rules

When can the experiment start?

Stop Criteria

When to end the experiment. Each early stop trades some power for a shorter run.

Stop at

Segments

Which subgroups should we track and break the results down by, to see if the effect differs across them?

Segment Your Bet Exclude Measure
Likely stronger: more to scroll past
Likely stronger: more to choose from

Iteration Rules

What next experiments to run once the results come in.

Outcome Re-run with: Higher intensity Higher power Elsewhere Something different Invert Move on
IF Stat sig primary A clear, significant win.
IF Stat sig primary Flat secondary A win on primary, no effect on secondary.
IF Flat primary Stat sig secondary No effect on primary, a win on secondary.
IF Flat primary Flat secondary No significant effect on either.
IF Stat sig neg primary A clear, significant loss.

Export to Experimentation Tools

Pick a destination — copy a plain-text spec, or a JSON config for your platform.


            
            
        

Now Run This Experiment