E-posta Kampanyası Gerçekten Satış Getirdi mi?Did the Email Campaign Really Drive Sales?

A campaign report usually says: “Customers who received the email spent this much.” But some of those customers would have bought even without the email. In this study I used an email campaign experiment, in which customers were split into random groups, to look at three questions: did the campaign really drive sales, who did it work for, and does a targeting model add anything?

Data and experiment

The data belongs to a US retailer and was shared by Kevin Hillstrom in 2008 for an open challenge (MineThatData). 64,000 customers who had bought in the last year were split at random into three groups: one received an email featuring men's products, one an email featuring women's products, and the third received nothing. Over the next two weeks, whether each customer visited the site, whether they bought and how much they spent were recorded. Also known: months since last purchase, last year's spend, whether they had bought men's or women's products before, whether they were a new customer, area type and purchase channel.

First, the experiment itself

Before looking at results I checked that the experiment was set up properly. The group sizes are very close (21,306, 21,307 and 21,387); a split like this is very likely to arise by chance (p = 0.90). The groups are also alike in customer characteristics: no standardised difference exceeds 0.014, where below 0.1 is generally considered balanced. So differences between the groups come from the email.

Did the campaign drive sales?

Yes, both emails worked. The men's email raised the share of customers visiting the site by 72%, more than doubled the share who bought, and added $0.77 of spend per customer. The women's email had a smaller effect: $0.42 per customer.

Not all of the spend belongs to the email

Customers who received no email also spent $0.65 each over the two weeks. Of the $1.42 spent by men's email recipients, only 54% is the email's effect. A campaign report crediting all of the recipients' spend to the campaign would overstate the effect 1.8 times; 2.5 times for the women's email.

This is the most common error in campaign measurement without an experiment. Without a control group, “would these sales have happened anyway?” cannot be answered.

Where does the effect come from?

More customers bought; those who bought did not spend more. The average basket of buyers who received the men's email was $113.53, against $114.00 for those who received nothing. This comparison rests on only 267 and 122 buyers, so it cannot see a small difference, but there is no sign of bigger baskets.

Which email is better?

The men's email brought $0.35 more spend per customer (interval $0.03 – $0.66, p = 0.03). There is a difference, but the experiment can only just see it. With spend this volatile, detecting a difference of this size with 80% probability needs about 28,974 customers per group; this experiment has 21,307. If the comparison were repeated, it would not be surprising for the difference to disappear.

Who did it work for?

I split customers into three by the products they had bought before. Here I look at the visit rate, because spend is too noisy within subgroups.

• The women's email barely worked on customers who had bought only men's products: visit rate +1.1 pts. For customers who had bought only women's products the effect was +7.4 pts. The gap is too large to be chance (p < 0.001).
• The men's email worked equally well in both groups: +6.9 pts and +7.1 pts.

Spend shows a difference in the same direction, but its interval includes zero. For other customer characteristics the picture is less clear:

Almost all intervals are wide and overlap. What this data says is that the men's email works to roughly the same degree for almost everyone.

Does a targeting model add anything?

A common question in campaigns: “Instead of sending to everyone, what if a model chose which email each customer gets?” To test this I built a separate gradient boosting model for each group (a T-learner). For each customer the model predicts the outcome of all three options and picks the best. Every customer was scored by models that had not seen them (5-fold cross-fitting).

Because the groups are random, each policy's result can be measured directly: it is the average of the customers who happened to receive the email the policy recommends.

• One of the best options is the simplest: the men's email to everyone. The product-category rule cannot be told apart from it (difference +$0.006, interval −$0.18 – $0.20).
• The models are not better. The model that predicts spend is significantly worse on visits than the men's email for everyone (−0.9 pts); it took noise for pattern.

For visits the model is slightly better than a random order, for spend it is not: mailing 30% of customers would capture only 19% of the spend effect. When the effect is spread this evenly across customers, a targeting model has nothing to sort out.

Is the campaign profitable?

The data has no email cost or profit margin. So I asked: what is the highest cost per email at which the campaign does not lose money? The answer is the extra spend times the gross margin.

For example, at a 30% margin the men's email is profitable at any cost below $0.23 per customer. The direct cost of sending an email is usually far below that; the real cost may be customers who unsubscribe or start ignoring the emails. Because this experiment covers two weeks, it cannot see that.

Limitations

Conclusion

The campaign worked, but about half as well as a report would show. Without a control group, purchases the recipients would have made anyway would have been credited to the campaign.

The second lesson is about targeting. A complex model could not beat the rule “send everyone the same email”, because the effect hardly varied between customers. The question to ask before the model is whether there is any difference to sort out.

Data: Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge (2008). Amounts in US dollars.