Dunwell Insights · Campaign Impact Check · Sample
Both emails sold more. No model beat sending to everyone.
Buyers per 1,000 customers in the two weeks after the email
Would have bought anywayExtra buyers the email caused
Summary for the owner
Send the men's email to everyone. Don't pay for targeting yet.
What we found
What we'd do
Open the live dashboardChoose how many customers to email, compare every targeting rule, change the margin and email cost, and download the send list.1 · The test
A clean random test: three equal groups that looked the same before the email
Customers in each group
The groups matched on everything known before the email
Standardised difference from the no-email group, every customer detail. Under 0.1 is the usual sign of a fair split.
2 · Did the emails work?
Share who bought, with its likely range
95% interval for each group
The email mostly brings visits. Few extra visitors buy.
Change caused by each email, percentage points
Every result, email against no email
3 · Who responds more?
The men's email worked across the board. The women's email looked weaker for established customers.
Extra purchases by what customers bought last year
Men's emailWomen's email
Women's email: new customers against established ones
Men's emailWomen's email
How we kept chance patterns out
4 · Can a model pick who to email?
Buyers per 1,000 customers, by how many you email
Gain over emailing random customers
Extra buyers per 1,000 customers, averaged over emailing 10% to 60% of the list. Tested on the 30% hold-out, with 95% range.
The order of the models flipped on new data
Rank in training (cross-fitted) against rank on the hold-out
Predicted gain against what really happened
Seen in the hold-out (men's email)What the model predicted
Likely buyers or persuadable buyers?
Emailing 30% of the hold-out, two ways
5 · What it's worth
These figures use a margin and an email cost we assumed. They are not in the data. The dashboard lets you put in your own. The window is two weeks, so this is not lifetime value, and the test did not record unsubscribes.
6 · Next steps
Send to everyone, keep a hold-back, and run the targeting test properly
7 · How we did this
The plan was written down before any result was seen
The rules for this study were locked before we compared any outcomes. The targeting rule was frozen, with a fingerprint of its scores, before the hold-out was opened. If something had changed after that, the run would have stopped. The whole study reruns from the raw file with one command.
The six models
Checks we ran
Decisions log
What this data can't tell us
- It is one email test from 2008 at one US retailer. It says nothing direct about discounts, paid ads, text messages or shops today.
- Results cover two weeks after the email. Longer effects, repeat orders and lifetime value were not measured.
- There is no margin, email cost or unsubscribe data. Every money figure rests on stated assumptions.
- No individual's response is ever observed under both emails. Model scores are estimates for groups of similar customers, not facts about a person.
- "No model beat random" means this test was too small to show a gain. It does not prove targeting can never work.
- The data has no customer ID, so the send list uses a row reference, not real people.
This analysis is based on the data described here and the methods shown. It supports decisions. It is not a promise of any result, and it is not financial, legal or tax advice.
Every job. Done well.