A restaurant marketing holdout test compares guests who are eligible to receive a campaign with similar guests randomly assigned not to receive it. The difference in their return rates helps estimate how many visits the campaign caused, rather than counting every returning guest as a campaign success.
That difference matters because some guests would have returned without an offer. A useful test also accounts for rewards given to those guests, campaign costs, and the contribution from the extra visits. You do not need a complicated dashboard to start, but you do need a consistent guest list and a way to count visits in both groups.
Choose eligible guests and a random holdout
Start with a business decision. For example: should the restaurant send a return-visit reward to guests who have not visited recently? Define the audience before splitting it. You might choose guests whose last recorded visit was 45 to 90 days ago, excluding employees and anyone already included in another targeted offer.
Use the same eligibility rules for both groups. Access to a guest's contact details is not the same as permission to send marketing. Check your permission records and applicable requirements before sending; ask a qualified adviser about legal questions. Decide how you will handle guests who cannot be contacted on the chosen channel before the random assignment, rather than removing them afterward only from the campaign group.
Give each eligible guest a random assignment to one of two groups. The treatment group is eligible to receive the campaign. The holdout group receives no version of that campaign during the observation window. Random assignment helps distribute frequent visitors, occasional visitors, and other guest characteristics across the groups. It does not guarantee perfectly matched groups, especially when the list is small.
Split by guest, not by phone number in one list and email address in another. If the same person appears twice, deduplicate the records before assignment where possible. Otherwise, a guest could land in both groups and receive the offer despite being counted as a holdout.
Document the assignment date, eligibility rules, group sizes, offer, send date, and planned end date. Keep guests in their original groups for the main analysis, including treatment guests who did not open the message or redeem the offer. Removing non-openers after the send selects a more engaged audience and can exaggerate the campaign's effect.
Your normal business does not have to stop. Both groups can still see a public menu or walk past the restaurant. The important distinction is that the holdout should not receive the targeted campaign being measured. If a broad promotion reaches both groups, record it; it may affect how you interpret the result.
Set the return window and count one outcome
Choose a primary outcome before sending. A practical choice is the share of assigned guests who make at least one identified purchase during the 28 days after the campaign send date. Count a guest once for this outcome, even if they visit twice. That measures returning guests, not total transactions or total sales.
Define an identified purchase consistently. If your records can connect an online order to a guest but cannot reliably connect an anonymous counter purchase, say so in the test notes. Apply the same rules to treatment and holdout guests. A return you cannot connect to a guest will not appear in either group's measured rate.
Use the same start and end dates for both groups. If the offer expires halfway through the window, keep watching until the planned end date. Otherwise, you may miss guests who return later without redeeming. Conversely, do not extend the window after seeing an unfavorable result just to look for a better number.
Before launch, check whether the systems used for ordering and guest records can give you the group-level counts you need. Ask a vendor how to export guest assignments, send history, redemptions, and identified purchases, and how records are matched across channels. An online ordering system can be part of the record, but your test definition should also say what happens to purchases made elsewhere.
For a practice run, use dummy guest records and simulated orders in a test environment. Confirm that assignment and counting work without triggering a real guest message, card payment, or courier dispatch. A live pilot is different: it sends the approved campaign to real eligible guests and measures actual activity. Review the setup and permissions before moving from simulation to a live send.
Work through an incrementality worksheet
The following figures are hypothetical. Suppose a restaurant randomly assigns 200 eligible guests to a campaign and 200 to a holdout. The campaign offers a $2 reward on a return purchase. The primary outcome is at least one identified purchase within 28 days.
The worksheet compares the groups first, then converts the estimated extra returning guests into an operational contribution. The $12 per incremental return is an assumed contribution after the direct costs of serving that purchase, but before the reward and campaign cost. It is not the sale amount or profit for the whole restaurant.
| Measure | Treatment | Holdout | Decision calculation |
|---|---|---|---|
| Assigned guests | 200 | 200 | Equal-sized groups |
| Guests with a return purchase | 40 | 26 | 40 minus 26 = 14 estimated incremental returning guests |
| Return rate | 20% (40 of 200) | 13% (26 of 200) | 7 percentage-point difference |
| Contribution before reward | 14 estimated extra returns at $12 each | Baseline reflected in the comparison | $168 estimated incremental contribution |
| Reward and campaign costs | 40 rewards at $2, plus $30 campaign cost | No test reward or campaign send cost | $80 + $30 = $110 |
| Estimated net contribution | $168 before test costs | Used as the baseline | $168 minus $110 = $58 |
The equal group sizes make the first calculation straightforward. If groups are different sizes, compare return rates first. Multiply the rate difference by the number of treatment guests to estimate incremental returning guests. For example, a seven-percentage-point difference applied to 200 treatment guests gives 14, not 14% of 200.
The example assumes all 40 returning treatment guests redeem one $2 reward, including people who would have returned anyway. Charging rewards only against the 14 estimated extra returns would understate the offer cost. It also assumes no holdout guest receives the test reward, no repeat purchase beyond the first is included, and the $30 covers the campaign's sending and setup costs. Payment fees and any other direct purchase costs are included in the assumed $12 contribution; they are not added again. Taxes, longer-term guest value, and other overhead changes are excluded.
Under those assumptions, the estimated $58 is positive. It is not proof that the campaign reliably earns $58 every time. If the restaurant's contribution estimate were $7 instead of $12, the calculation would be 14 × $7 = $98, then $98 − $110 = −$12. That would change the immediate decision: revise the offer or its cost before repeating it at scale.
Use your own contribution figure rather than average check size. Sales are what guests spend; contribution is what remains after the direct costs you choose to include. Deposits, if relevant to a different campaign, are cash collected in advance and should not automatically be treated as extra sales or contribution. Write down which costs are included so the next comparison uses the same basis.
Interpret the result without overstating it
A positive difference is an estimate, not a guarantee. With small groups, a few guests can move the rate substantially. Avoid declaring a winner from one short run simply because the treatment rate is higher. Plan a larger or repeated test when the decision and available audience justify it, and consider statistical help if you need to judge uncertainty more formally.
Look for events that could make the comparison harder to interpret. A local festival, a weather closure, a menu change, or a competing promotion can affect visits during the window. Events that reach both groups may still change the setting in which the offer performs. A targeted promotion sent disproportionately to one group is more concerning because the measured difference could reflect both campaigns.
Watch for selection bias. Comparing guests who redeemed an offer with guests who did not is not the same as comparing randomly assigned groups. Redeemers may already be more interested in returning. Likewise, comparing this month's campaign recipients with last month's guests mixes the offer with changes in season, schedule, and other conditions.
Check data quality before acting. Are duplicate guests counted twice? Can staff identify purchases in both groups in the same way? Did anyone in the holdout receive the targeted offer through another list? Record problems and their likely direction rather than quietly changing the denominator. If the records cannot support a fair comparison, fix the measurement and run another test.
Finally, distinguish a short-window result from long-term value. A reward may move a visit forward that would otherwise have happened later. A 28-day test cannot by itself show whether that guest made an additional purchase over several months. If that question matters, plan a longer follow-up and its outcome before looking at the later results.
Turn the result into a buying and operating decision
Use the test to decide what to do next, not merely to report an open rate. A positive estimate after costs may justify another controlled send. A negative estimate may point to a smaller reward, a lower campaign cost, or a different eligible audience. Change one major element at a time when practical, and keep a holdout so you can compare the next version.
When comparing software, ask whether your team can create and preserve randomized groups, suppress the holdout from the specific send, export the assignments, and match purchases back to guests. Also ask how the system handles duplicate records, consent status, redemptions, and multiple channels. Request a demonstration using dummy data rather than assuming a segment or redemption report is a complete incrementality report.
Fleksa's marketing tools include email and SMS campaigns, segmentation, promo codes, gift cards, and loyalty points for eligible orders. Ask how your intended test would be configured and measured in your setup. For a broader list of purchasing considerations, see the marketing platforms buying guide; for the operating context, review restaurant management software.
Next, write down your eligible audience, primary outcome, observation window, and expected reward cost. Then compare those requirements with Fleksa's restaurant marketing automation and ask for a walkthrough of the assignment, suppression, and export steps.
Frequently asked questions
What is a restaurant marketing holdout test?
It is a comparison between randomly assigned eligible guests who can receive a campaign and guests who do not receive that campaign. The difference in their outcomes estimates the campaign's incremental effect.
Should I count only guests who redeem the offer?
No. For the main comparison, keep all assigned guests in their original groups and count the defined return outcome. Separately track redemptions to calculate reward cost.
What if my holdout group returns more often?
The observed result does not support an immediate positive lift for that run. Check execution and data quality, consider sample uncertainty, and decide whether a revised offer or another controlled test is worthwhile.
