Test Priority
by
Test prioritization for ecommerce teams: size each idea against the traffic it needs and the revenue it could move, then write the brief and the hypothesis.
Most test backlogs are ordered by whoever argued hardest, and most tests never conclude. The two problems are the same problem: nobody checked whether the page had the traffic to detect the effect being looked for. A test on a page with four hundred weekly sessions needs an implausibly large effect to reach significance, so it runs for six weeks and ends inconclusive, and the team learns nothing except that testing is slow. This skill sizes each idea against the traffic it needs and the revenue it could move, then writes the brief and the hypothesis.
What It Does
- Estimates the minimum detectable effect each page can support at its traffic.
- Sizes the revenue at stake behind each idea before ranking it.
- Rejects the tests that cannot conclude in a useful timeframe.
- Writes a hypothesis specific enough to be wrong.
- Specifies what to measure and what to leave alone.
- Generates the brief, the priority and the hypothesis for each test.
Use Cases
- Stop running tests that mathematically cannot conclude.
- Order a backlog by revenue at stake rather than by enthusiasm.
- Turn a vague idea into a hypothesis that can actually be falsified.
- Know the runtime before you start rather than six weeks in.
Example Session
Here is our test backlog. What should we run and in what order?
Three of these seven can conclude. Run them in this order.
- Test | Weekly sessions | Detectable effect | Revenue at stake | Runtime | Rank
- Checkout, remove address line 2 | 18,400 | 3.1% | $146,000 | 11 days | 1
- PDP, move reviews above fold | 9,200 | 5.8% | $88,000 | 19 days | 2
- Homepage hero copy | 24,000 | 2.4% | $41,000 | 9 days | 3
- Category page filters | 1,100 | 22% | $12,000 | Never | Do not run
The category filter test needs a 22% swing to register at eleven hundred sessions a week, which no filter change has ever produced. It cannot conclude, so it should be shipped on judgment or dropped rather than tested. The checkout test is first because $146,000 is at stake and eleven days is short enough that the result still matters when it lands.
Why This Beats Prompting It Yourself
An Interesting Idea vs A Test That Can Conclude: The Hidden Risk
A generic prompt evaluates test ideas on plausibility and effort, which are the two things everyone in the room can already judge. It has no view of the traffic each page gets, so half the resulting backlog consists of tests that will run for weeks and end with a shrug. The team concludes testing does not work for them.
When You Still Need Permission To Just Ship It
A change that cannot be tested is not automatically a bad change. The category filter improvement may well be right, and this skill says so: ship it on judgment rather than pretending an underpowered test will settle it. Knowing which decisions testing can actually resolve is what keeps a program credible.
Known Limitations
Detectable effect depends on the baseline conversion rate and variance you supply, and a page with unusually noisy traffic will need more than the estimate suggests. Revenue at stake assumes the effect applies to the whole segment tested, which overstates tests on a narrow page. It writes the brief and the hypothesis, and does not build or run the test.
Install It
- Download the ZIP.
It is free and there is no account to create. - Unzip it into your agent's skills folder.
Claude Code reads ~/.claude/skills/, which is hidden by default: the command in the folder block opens it. Other agents scan their own directory, so drop the same folder there instead. - Ask your agent to use it.
Restart the agent if it was already running, then it picks the skill up with no config.
skills folder
Copy
~/.claude/skills/test-priority/
SKILL.md
references/power-model.md
briefs/
# one brief per test, with the hypothesis
# macOS: create the folder and open it in Finder
mkdir -p ~/.claude/skills && open ~/.claude/skills
# Windows: paste in the Explorer address bar
%USERPROFILE%\.claude\skills\Frequently asked questions
How is this different from a prioritization framework?
Most frameworks score ideas on impact, confidence and effort, all of which are opinions. This computes what each page can actually detect at its traffic and sizes the revenue behind it, which turns two of the three into measured quantities and rules out the tests that cannot finish.
Which agents does the skill run in?
Any agent that supports the open SKILL.md format: Claude Code, Cursor, Codex CLI, GitHub Copilot, Gemini CLI, Manus, Grok and others load it unmodified. The format is portable, the location is not. Each agent scans its own skills directory, so you drop the same folder into whichever one yours uses. There is no config file to edit and no API key to provision.
What does it need connected to work?
Page-level traffic and conversion, order data, and your backlog. It does not need a testing tool connected, and the briefs it writes work in whatever tool you already use.
Why refuse to run a test rather than just running it slowly?
Because a six week inconclusive test costs six weeks and produces no decision, and running it longer usually means the site changed underneath it. The honest options are ship on judgment, or find a higher traffic surface to test the same idea on.
What makes a hypothesis good enough?
It has to be capable of being wrong. "Improve the PDP" cannot fail. "Moving reviews above the fold raises add-to-cart on mobile by at least 5.8%" either happens or does not, and the number comes from what the traffic can detect.
What is included with the skill?
The skill itself, the power model reference, the brief format, and the hypothesis template. It is free and security scanned. We re-publish the ZIP when the model changes, so download it again if your runtimes start shifting.
Test priority is a revenue ranking, not a wish list.
Polar sizes each opportunity in net revenue on one definition, so the brief at the top of the list is the one worth the sprint.
<script type="application/ld+json">{"@context":"https://schema.org","@type":"FAQPage","mainEntity":[{"@type":"Question","name":"How is this different from a prioritization framework?","acceptedAnswer":{"@type":"Answer","text":"Most frameworks score ideas on impact, confidence and effort, all of which are opinions. This computes what each page can actually detect at its traffic and sizes the revenue behind it, which turns two of the three into measured quantities and rules out the tests that cannot finish."}},{"@type":"Question","name":"Which agents does the skill run in?","acceptedAnswer":{"@type":"Answer","text":"Any agent that supports the open SKILL.md format: Claude Code, Cursor, Codex CLI, GitHub Copilot, Gemini CLI, Manus, Grok and others load it unmodified. The format is portable, the location is not. Each agent scans its own skills directory, so you drop the same folder into whichever one yours uses. There is no config file to edit and no API key to provision."}},{"@type":"Question","name":"What does it need connected to work?","acceptedAnswer":{"@type":"Answer","text":"Page-level traffic and conversion, order data, and your backlog. It does not need a testing tool connected, and the briefs it writes work in whatever tool you already use."}},{"@type":"Question","name":"Why refuse to run a test rather than just running it slowly?","acceptedAnswer":{"@type":"Answer","text":"Because a six week inconclusive test costs six weeks and produces no decision, and running it longer usually means the site changed underneath it. The honest options are ship on judgment, or find a higher traffic surface to test the same idea on."}},{"@type":"Question","name":"What makes a hypothesis good enough?","acceptedAnswer":{"@type":"Answer","text":"It has to be capable of being wrong. \"Improve the PDP\" cannot fail. \"Moving reviews above the fold raises add-to-cart on mobile by at least 5.8%\" either happens or does not, and the number comes from what the traffic can detect."}},{"@type":"Question","name":"What is included with the skill?","acceptedAnswer":{"@type":"Answer","text":"The skill itself, the power model reference, the brief format, and the hypothesis template. It is free and security scanned. We re-publish the ZIP when the model changes, so download it again if your runtimes start shifting."}}]}</script>