Holdout groups: how to run one and what it will tell you

Every flow in your account is credited with revenue, and a large share of that revenue would have happened anyway. This is not a flaw in the tooling — attribution honestly reports which message preceded the order. It simply cannot answer the different and more important question of whether the message caused it.

A holdout is the answer, and it is unglamorous: withhold the message from a random slice, and compare.

How to run one

1. Split randomly at the person level, not by segment, region or alphabet. Anything non-random imports the difference you were trying to measure.
2. Hold the split stable over the test. Re-randomising each send makes the comparison meaningless.
3. Measure revenue per person across both groups over a fixed window after entry, not revenue per message — the held-out group received none.
4. Include everybody who qualified, whether or not the message was delivered. Excluding bounces from one side and not the other biases the result.
5. Report the difference, which is the caused revenue. The attributed number stays on the dashboard; this one goes in the decision.

Sizing it, roughly

Two rules of thumb that avoid the statistics:

• Continuous flows: hold out 10 % permanently. Volume accumulates, and after a few thousand entries per side the difference is readable.
• One-off campaigns: 10 % of a single send is almost never enough to detect anything. Either hold out a much larger share once a quarter, or accept that campaigns are measured relatively against each other rather than absolutely.

If the difference you are trying to detect is small and your volume is small, the honest conclusion is that you cannot measure it, which is itself worth knowing before you spend six months optimising it.

Where it is worth running permanently

1. Abandoned cart. The flow most flattered by attribution, because everybody in it was already mid-purchase. Expect the true lift to be a fraction of the attributed number — still positive, usually still worth running, and much smaller than the dashboard says.
2. Browse abandonment. Often the one where the honest answer is near zero, which is a real finding worth acting on.
3. Win-back. A meaningful share of lapsed customers return on their own, and the tiers usually differ sharply from each other.
4. Replenishment. Frequently the biggest true lift in the whole programme, because the message arrives at a moment the customer would otherwise have to remember unaided.

Reading the result, including zero

Three outcomes, and the middle one is the one people mishandle.

• Clear positive lift. Keep it, and consider whether it should run more often or to a wider audience.
• No detectable difference. This does not mean no effect — it means no effect you can measure at this volume. Decide on cost: a flow that costs nothing to run can stay; one that costs per message should stop.
• Negative. Rarer, real, and usually a frequency problem — the message is displacing something better, or pushing people to unsubscribe. Check unsubscribes in both arms before concluding anything else.

Two practical objections, answered

"We are losing revenue by withholding." You are forgoing 10 % of whatever lift exists, in exchange for knowing the size of the lift on the other 90 %. If the flow is worth running, that is a small price; if it is not, the holdout is what tells you to stop paying for it.

"It is unfair to the held-out customers." They receive everything else you send. And no holdout should ever withhold a transactional message, a consent confirmation or anything a customer is entitled to — only marketing they never knew was coming.