What is holdout testing?
A holdout test is a randomized controlled trial for marketing. Before a campaign runs, people or places are split at random. One group gets the campaign, and the holdout doesn't. Afterward, the difference in outcomes between them is the campaign's incremental effect.
It's the most direct way to answer whether marketing caused the results credited to it. Attribution shows who converted after an ad. A holdout shows how many would have converted without it.
How do you run a holdout test?
- Pick one question. Choose the channel, campaign, or tactic, and the outcome that matters, such as orders or new customers.
- Choose the unit. Split by user when the platform can withhold ads reliably, or by region for channels like TV, out-of-home, and retail.
- Size the holdout. Work out the smallest lift worth detecting and the sample needed to see it. Smaller effects need bigger or longer tests.
- Run it clean. Keep the groups separate for the full test, and don't change budgets or targeting halfway through.
- Read the lift. Compare outcomes, report the range around the estimate, and turn it into incremental cost per result.
Many ad platforms offer conversion lift studies that run user-level holdouts inside their own systems. They're convenient, but they measure that platform's ads with that platform's data, so many teams pair them with geo-level tests they control.
What is an example of a holdout test?
Testing an email program
A retailer sends a weekly promotional email to two million customers. Its reports credit the email with 8% of online revenue.
For six weeks, a random 10% of the list receives nothing. That group buys only slightly less than the rest. The email's true lift is about a quarter of what reports showed, because most of the people who bought after opening it were going to buy anyway.
Holdout tests vs. geo-lift tests vs. A/B tests
| User holdout | Geo-lift test | A/B test | |
|---|---|---|---|
| What's compared | Exposed vs. unexposed people | Treated vs. untreated regions | Version A vs. version B |
| Question it answers | Did the marketing work at all? | Did the marketing work at all? | Which version works better? |
| Best for | Digital channels with user-level control | Channels that can't target people, such as TV | Creative, offers, and landing pages |
| Main risk | Leakage between groups | Regions that drift apart | Testing only versions of something that may not work |
Why does holdout testing matter?
Every other measurement method in marketing is estimating what a holdout measures directly. That's why holdouts are used to calibrate mix models and to check attribution.
- A real baseline. The holdout shows what would have happened, so the lift isn't a guess.
- Calibration. Test results anchor models, so their estimates stay close to what the tests found.
- A known cost. The holdout gives up some sales during the test. That cost is the price of knowing, and it's usually far smaller than the cost of funding a channel that doesn't work.
How does Alli do this?
Holdout tests and modeling answer different questions. A test measures one channel at one point in time, and a model compares every channel continuously. Alli Scenario Planner covers the modeling side, pairing a media response model built on historical sales with signals from Alli Brand Equity.
Teams test budget shifts in a live environment, see where each channel's returns start to diminish, and compare scenarios before committing spend. Forecasts are modeled estimates, and many teams keep their own holdout tests alongside them.
Explore Alli Scenario PlannerHoldout testing FAQ
What is a holdout group in marketing?
A holdout group is a random set of people or places that don't receive a campaign during a test. Comparing them with the group that did shows the campaign's true effect.
How big should a holdout group be?
Big enough to detect the smallest lift worth acting on. That depends on your conversion rate, how much results vary, and how long the test runs. Many tests hold out 5% to 20%.
How long should a holdout test run?
Long enough to cover a full buying cycle and reach the needed sample, often two to six weeks. Stopping early when results look good inflates the lift.
Is a holdout test the same as an A/B test?
No. A holdout test compares marketing with no marketing. An A/B test compares two versions of the marketing. Both use random assignment.
Last updated Back to the glossary