Incrementality
by
Holdout testing for ecommerce teams: design a test that can actually conclude, then read the real lift when it lands rather than the platform's version.
Every platform reports the conversions it can claim, and the sum of those claims exceeds what the business sold. The only way past that is a holdout: withhold spend from a comparable group and measure what changes. Most attempts fail for the same two reasons, a holdout too small to detect the effect and a duration too short to cover the purchase cycle, so the result is ambiguous and everyone returns to the platform numbers. This skill designs a test that can conclude, then reads the real lift when it lands.
What It Does
- Sizes the holdout needed to detect the effect you care about.
- Sets the duration against your actual purchase cycle, not a round number.
- Picks a holdout group comparable on the dimensions that matter.
- Estimates the revenue the test itself will cost while it runs.
- Guards against contamination from other campaigns reaching the holdout.
- Reads the lift when the test lands, with the confidence interval.
Use Cases
- Find out what a channel is really contributing rather than claiming.
- Size a holdout that can actually detect the effect you care about.
- Price the cost of running the test before committing to it.
- Read a completed test without reading into the noise.
Example Session
We want to know if our brand search spend is incremental. Design the test.
Hold out 18% of geos for 34 days. The test costs about $14,000 to run.
- Parameter | Value | Why
- Holdout size | 18% of geos | Detects a 12% lift at your volume
- Duration | 34 days | Covers your 26 day purchase cycle plus buffer
- Cost of running | $14,000 | Revenue forgone in held-out geos
- Detectable effect | 12% | Smaller effects need a longer test
Eighteen percent is the smallest holdout that can detect a twelve percent lift at your volume, and twelve percent is the threshold where the answer would change what you do. Thirty-four days rather than thirty because your purchase cycle runs twenty-six days and a shorter test would measure customers still deciding. The $14,000 is the real cost of knowing, and brand search is currently taking $18,000 a month on an unproven claim.
Why This Beats Prompting It Yourself
Platform Attribution vs A Holdout: The Hidden Risk
A generic prompt compares the platform's reported conversions against your orders and concludes something is overstated, which is true and not actionable. Attribution models disagree by design and no amount of comparing them settles the question. Only withholding spend and watching what happens produces an answer, and that has to be designed before it can be run.
When You Still Need The Test's Own Cost Priced
A holdout costs money while it runs, and that number is what makes the decision real. Fourteen thousand dollars to find out whether eighteen thousand a month is doing anything is an obviously good trade. The same test on a channel spending two thousand a month is not, and this skill says so rather than designing it anyway.
Known Limitations
Geo-based holdouts assume regions are comparable enough that differences reflect the treatment, which breaks where your customer base is concentrated in a few cities. Contamination from organic, email and other channels reaching held-out geos is controlled for where possible and flagged where not. It designs and reads the test; withholding the spend is something you do in the platforms.
Install It
- Download the ZIP.
It is free and there is no account to create. - Unzip it into your agent's skills folder.
Claude Code reads ~/.claude/skills/, which is hidden by default: the command in the folder block opens it. Other agents scan their own directory, so drop the same folder there instead. - Ask your agent to use it.
Restart the agent if it was already running, then it picks the skill up with no config.
skills folder
Copy
~/.claude/skills/incrementality/
SKILL.md
references/holdout-design.md
tests/
# one file per test, with the design and the read
# macOS: create the folder and open it in Finder
mkdir -p ~/.claude/skills && open ~/.claude/skills
# Windows: paste in the Explorer address bar
%USERPROFILE%\.claude\skills\Frequently asked questions
How is this different from comparing attribution models?
Comparing models tells you they disagree, which you already knew. A holdout measures what actually changes when the spend stops, which is the only evidence that settles the question. This designs one that can conclude and reads it when it does.
Which agents does the skill run in?
Any agent that supports the open SKILL.md format: Claude Code, Cursor, Codex CLI, GitHub Copilot, Gemini CLI, Manus, Grok and others load it unmodified. The format is portable, the location is not. Each agent scans its own skills directory, so you drop the same folder into whichever one yours uses. There is no config file to edit and no API key to provision.
What does it need connected to work?
Spend and revenue by geo, order data with purchase timing, and the channel structure. Purchase cycle length is derived from your own order data and is what sets the duration.
Why does the holdout need to be that large?
Because a smaller one cannot distinguish the effect from normal variation, and an underpowered holdout produces an ambiguous result that everyone interprets in whichever direction they already believed. Eighteen percent is the smallest size that answers the question.
What if the test comes back inconclusive?
The skill reports it as inconclusive with the interval, rather than reading a direction into noise. Where the interval is wide but centered somewhere useful it says what a longer test would resolve and what it would cost.
What is included with the skill?
The skill itself, the holdout design reference, the test format, and the read-out method. It is free and security scanned. We re-publish the ZIP when the method changes, so download it again if your designs start shifting.
A holdout is only as good as the baseline behind it.
Polar reconciles 45+ sources into one definition of net revenue, so the lift you read when the test lands is lift, not a reporting artifact.
<script type="application/ld+json">{"@context":"https://schema.org","@type":"FAQPage","mainEntity":[{"@type":"Question","name":"How is this different from comparing attribution models?","acceptedAnswer":{"@type":"Answer","text":"Comparing models tells you they disagree, which you already knew. A holdout measures what actually changes when the spend stops, which is the only evidence that settles the question. This designs one that can conclude and reads it when it does."}},{"@type":"Question","name":"Which agents does the skill run in?","acceptedAnswer":{"@type":"Answer","text":"Any agent that supports the open SKILL.md format: Claude Code, Cursor, Codex CLI, GitHub Copilot, Gemini CLI, Manus, Grok and others load it unmodified. The format is portable, the location is not. Each agent scans its own skills directory, so you drop the same folder into whichever one yours uses. There is no config file to edit and no API key to provision."}},{"@type":"Question","name":"What does it need connected to work?","acceptedAnswer":{"@type":"Answer","text":"Spend and revenue by geo, order data with purchase timing, and the channel structure. Purchase cycle length is derived from your own order data and is what sets the duration."}},{"@type":"Question","name":"Why does the holdout need to be that large?","acceptedAnswer":{"@type":"Answer","text":"Because a smaller one cannot distinguish the effect from normal variation, and an underpowered holdout produces an ambiguous result that everyone interprets in whichever direction they already believed. Eighteen percent is the smallest size that answers the question."}},{"@type":"Question","name":"What if the test comes back inconclusive?","acceptedAnswer":{"@type":"Answer","text":"The skill reports it as inconclusive with the interval, rather than reading a direction into noise. Where the interval is wide but centered somewhere useful it says what a longer test would resolve and what it would cost."}},{"@type":"Question","name":"What is included with the skill?","acceptedAnswer":{"@type":"Answer","text":"The skill itself, the holdout design reference, the test format, and the read-out method. It is free and security scanned. We re-publish the ZIP when the method changes, so download it again if your designs start shifting."}}]}</script>