RCT or A/B test? How to pick the right test for your behaviour question

Lauren Kelly

An A/B test is quick, light and brilliant for micro‑tweaks.
An RCT is slower and stricter but protects you from costly false wins.
Pick the wrong method and you either waste speed or budget, or risk credibility.

Know the tools

A/B test

What it is:

One moment in time, one audience pool. You randomly show Version A to half and Version B to the other half, then compare what they do next.

Think of it as:

A split‑screen coin toss. Two doors, same corridor, same day. Everything apart from the door label is identical.

Use when:

Context is stable (no season change, no big ads dropping).
The decision is cheap or reversible (button colour, headline).
You need a quick signal, not courtroom‑level proof.

RCT (Randomised Controlled Trial)

What it is:

A structured experiment that runs over time or across places. Participants (or whole sites) are randomly put into Control or Treatment groups. You track both groups before, during, and after the change to see the true effect while filtering out seasonality, policy shifts, and outside noise.

Think of it as:

A medical drug trial for behaviour. Two wards, identical patients, one gets the pill, one gets the placebo, and you watch for weeks.

Use when:

Stakes are high (money, safety, reputation).
The context could move under your feet (holidays, new policy, multi‑country rollout).
Finance, regulators, or the press will challenge the evidence.

Key Differences

Draw the Research Decision Matrix

Plot Decision cost on one axis and Confidence needed on the other.

Bottom‑left box: Cheap choice, low confidence. A/B is enough.
Top‑right box: Expensive or sensitive choice, ironclad proof needed. Bring out the RCT.

Do this on a whiteboard before you write the research plan.

Write a test charter first

Objective: What behaviour will change and how will we measure it?
Risk: Money, reputation, user welfare.
Timeframe: From first exposure to meaningful outcome.
Ethical stakes: Data sensitivity, possible harm.

The charter stops you choosing a heavyweight method for a lightweight question and vice versa.

Check context stability

Ask: Will anything important change while the test runs?

Context is incredibly important to behaviour. Most of it comes from the things around us. So if it changes, it could rock your tests.

Look for:

Seasonality or campaign overlap
Multiple sites or cultures
Policy shifts or system upgrades

If you answer yes to any, treat this as a sign that an RCT is needed.

Things to avoid

Final checklist

What is the downside of a wrong answer?
Will context stay stable during the test?
Do regulators, finance or media need convincing?
Is the behaviour long tail or immediate?

If you tick yes for questions 2 to 4 and the downside is serious, choose an RCT. Otherwise do an A/B and move fast.

Pocket Rules

Default ≠ Dark Pattern. It turns dark when the exits hide or hurt.
Conversion without consent costs trust, and trust costs more to rebuild.
Would I feel tricked? If yes, so will your users. Change it.