Thinking · Newsletter
Incrementality testing for ad creative: the bake-off, the holdout, and what to hold constant
How to tell whether a new channel or a new creative set adds demand or re-counts the customers you already had, with the two mistakes that make a result unreadable.
TL;DR. Incrementality testing answers the only question that matters about paid media: did this cause sales that would not have happened anyway? For creative, the practical version is a two-step: a bake-off of native creative against your existing repurposed set on the new channel (tells you whether the creative reaches new people), then a holdout test once spend justifies it (tells you how much of the lift is real). Platform-reported ROAS answers neither. This page explains the method, what to hold constant, how long to run, and the two mistakes that make a result unreadable.
What is incrementality testing?
A method for measuring the sales or installs caused by an activity, rather than the ones attributed to it. Attribution hands credit to whichever touch the model chooses; incrementality compares a group that saw the activity with a comparable group that did not. The difference is the lift. For channel diversification, it is the only way to tell new demand from the same customers re-counted.
Why creative needs its own incrementality logic
Most incrementality tests measure a channel. But on the platforms that matter now the creative decides who the channel reaches, so two creative sets on the same channel can reach two different populations. That is why we run the bake-off first: it is a creative-fit test that doubles as an early incrementality signal.
The bake-off
Run your existing repurposed creative against creative built natively for the channel, at the same time, on the same channel, under the same conditions.
| Repurposed baseline | Native concepts | |
|---|---|---|
| Built for | The original channel | This channel |
| Tells you | Whether the channel works at all | Whether it adds demand |
| Runs | Same window | Same window |
If the native set merely reaches the audience the old creative already reached, it performs like the baseline. If it clearly beats the baseline, the channel is reaching people the old creative was not, which is incremental demand rather than reallocation. We ran this for HealthyCell on AppLovin: three native concepts against the repurposed Meta baseline, and the native set won on interaction and on purchase.
Hold constant: budget split, window, audience settings and the destination. Change only the creative. Minimum: three native concepts, each on a distinct hook, so a null result is informative rather than ambiguous. Duration: long enough to clear the platform's learning period and accumulate real purchase volume, not clicks.
The holdout
Once the bake-off says the channel is worth investing in, run a proper holdout: a geographic or audience split where one group is exposed and a comparable group is not, for long enough to read purchases. Measure the difference in the business metric, not in the platform's dashboard. This is what we sign to: holdout-measured creative lift, and share of spend the creative set captures, because the second is a fact about the auction rather than a claim about ourselves.
Two mistakes that make a result unreadable
- Changing two things. New creative and a new landing page in the same test teaches you nothing about either. Test the page in a second pair.
- Reading platform ROAS as the answer. Both platforms' self-reported numbers include customers who would have bought anyway. Use them to run the campaign, not to judge the channel.
Questions people ask
What is the difference between attribution and incrementality? Attribution assigns credit for a sale to touches a model chose. Incrementality measures sales that would not have happened without the activity, by comparing exposed and unexposed groups.
How do you run an incrementality test on AppLovin or Meta? Bake-off first (native versus repurposed creative, same channel, same window, only the creative changes), then a geo or audience holdout once spend justifies it. Read purchases in your own data.
How long does a creative incrementality test take? Past the platform's learning period and to meaningful purchase volume; for most mid-sized accounts that is weeks, not days.
What is a good incremental ROAS? One you measured. There is no benchmark that survives a different product, price and channel mix; the number only means something against your own baseline.
Sources: Flixr client work with HealthyCell, 2026; platform documentation on learning phases.