Running email, digital and direct mail campaigns in sync can increase your reach and boost your results. But it can also make attribution a bit tricky.

If a customer receives a retargeting postcard on Tuesday, opens an email on Thursday and converts after clicking a banner ad on Friday, which channel gets credit for moving the needle? The only method that produces incrementally accurate results is a structured test-and-control framework.

Here’s how to build one.

First, establish your baseline.

A baseline is an audience segment that receives no marketing, also referred to as a holdout group. Without it, your testing can only measure activity, not lift. Withholding marketing from a segment may feel counterintuitive, but it’s how you know what’s really driving results.

Holdout groups represent 10-20% of your total audience. The catch? The group must be randomly selected, so every individual has an equal probability of seeing and reacting to your marketing. Splitting a list in half isn’t a valid test design, and any pattern in how the holdout group is assembled introduces bias and undermines the entire test.

Next, build valid tests across channels.

Testing within a channel compares two versions of the same medium — think two direct mail creative executions, two email subject lines or two offer variations. The goal is learning which version performs better within a defined environment.

Cross-channel testing is more complex. Here, you’re comparing the performance of audience segments exposed to different channel combinations — for example, mail plus email versus email only, or all three channels versus digital only. These tests answer bigger strategic questions, but only if each cell is randomly assigned and isolated from the others.

For more mature programs, multi-variate testing allows you to examine multiple variables simultaneously, including offer type, creative format or channel combination within a single campaign. This requires large audiences, so it’s not the right starting point for every program.

Finally, put it all together.

For a campaign running across direct mail, email and social, a well-designed test structure might look like this:

• Test cell A. All three channels — mail, email and paid social
• Test cell B. Email and paid social only, no direct mail
• Test cell C (holdout). No marketing, establishing a true organic baseline

The gap between cell B and cell C tells you about the incremental lift from digital channels alone. The gap between cell A and cell B reveals the incremental contribution of direct mail specifically, justifying the investment or informing how to reallocate it.

Remember, none of this works without a documented test plan on what success looks like for each cell. Before launching any campaign:

• Define your primary metrics — conversion rate, revenue per piece, incremental ROAS, etc.
• Align your CRM, digital platforms and mail tracking systems so data flows into a unified view.
• Standardize attribution windows across channels.
• Run match-back reports on a defined schedule post-campaign.

Why direct mail has its own rules

Direct mail operates on different timelines and economics than digital channels, and your measurement framework needs to account for both.

Production and delivery timelines are longer. That means test cell design decisions (audience segmentation, holdout allocation and tracking setup) have to be finalized earlier in the planning process.

The higher per-piece cost of direct mail raises the stakes for smart test design. You can’t test every variable in every mailing. Prioritize the questions that will change how you allocate budget or approach creative. 

Attribution windows for direct mail diverge from digital norms. A display ad’s shelf life is measured in days, but a well-designed postcard or catalog can sit on a kitchen counter for weeks.

When building your tracking infrastructure, connect the physical and digital touchpoints using:

• Personalized URLs (PURLs) and dedicated landing pages to tie response back to specific mail recipients
• QR codes linking directly to trackable online destinations.
• Unique promo codes to attribute online conversions to specific mail segments
• Seeded lists to verify in-home delivery dates and help align attribution windows across channels

>>Related: How to Choose the Right Data-Driven Direct Mail Platform<<

Test-and-control methodology: The basics, answered

What is a test-and-control framework in marketing?

A test-and-control framework is a structured methodology for measuring the true incremental impact of your marketing campaigns. It works by dividing your audience into cells — some exposed to marketing, one withheld — so you can isolate what’s driving results.

How large should a holdout group be?

Holdout groups typically represent 10-20% of your total audience, though the right size depends on your overall list volume and the statistical confidence you need.

The more important factor is that the group is randomly selected as size alone doesn’t guarantee validity. A biased holdout, no matter how large, will produce unreliable results. 

What’s the difference between within-channel and cross-channel testing?

Within-channel testing compares two versions of the same medium, such as two subject lines or two direct mail creative executions, to find which performs better in isolation.

Cross-channel testing compares audience segments exposed to different channel combinations, answering bigger strategic questions about how channels work together. Both require random assignment, but cross-channel tests are more complex to design and analyze.

How do you measure the incremental value of direct mail?

The cleanest way is to run a test with one cell receiving all channels including direct mail, and another receiving only digital. The performance gap between those two cells represents the incremental contribution of direct mail. This approach isolates the channel’s impact rather than relying on last-click or match-back alone. 

Why is random selection so important in test design?

Random selection ensures every individual in your audience has an equal probability of being assigned to any given cell, which eliminates selection bias. If your holdout or test cells are assembled based on any pattern — geography, recency, purchase history — the results will reflect that pattern rather than your marketing.

How does direct mail attribution differ from digital attribution? Digital channels produce near-instant, trackable responses, but direct mail operates on longer delivery timelines and longer response windows. A display ad may drive action within hours, while a postcard or catalog can prompt a conversion days or weeks after it arrives.

Your attribution windows need to account for this difference, or you’ll systematically undercount direct mail’s contribution.

The real value of getting attribution right

Ready to build smarter measurement into your next multichannel campaign? Download our latest eBook: How to Crush Direct Mail, The Complete Guide for Marketers, and execute your next direct mail campaign successfully.