A/B Test

Any AB Test

Plan any A/B test: describe your change, then set up who sees it, what you measure, how long it runs and when it stops, with the same checks we use on every GoodUI template.

Lessons 7
With
defaults
from
Ron Kohavi Ex-Airbnb, Microsoft & Amazon Jakub Linowski Founder of GoodUI

Your Change

What B does differently, and why you expect it to win.

Experiment Design

Choose the structure of your experiment.

Targeting / Triggering

Refine where and when the test fires, beyond the page types chosen above in order to increase power (and shorten the testing duration).

Variance Reduction

Cancel out noise you can predict, to get more power from the same traffic.

Metrics & Ship Requirements

Mark each metric as Primary, Secondary, or Guardrail, then set the requirement it must meet to ship.

Metric Primary Secondary Guardrail Superiority Non-Inferiority Binary Continuous
%
%
%
%

Power, Sample & Duration

The effect you want to detect and how long you'll run to reach the power for it.

%
%
%
%

Total Sample — visitors across all variants
Duration —weeks  

Start Rules

When can the experiment start?

Stop Criteria

When to end the experiment. Each early stop trades some power for a shorter run.

Stop at

Segments

Which subgroups should we track and break the results down by, to see if the effect differs across them?

Segment Your Bet Exclude Measure

Iteration Rules

What next experiments to run once the results come in.

Outcome Re-run with: Higher intensity Higher power Elsewhere Something different Invert Move on
IF Stat sig primary A clear, significant win.
IF Stat sig primary Flat secondary A win on primary, no effect on secondary.
IF Flat primary Stat sig secondary No effect on primary, a win on secondary.
IF Flat primary Flat secondary No significant effect on either.
IF Stat sig neg primary A clear, significant loss.

Export to Experimentation Tools

Pick a destination — copy a plain-text spec, or a JSON config for your platform.


            
            
        

Now Run This Experiment