Holdback

📘

Holdbacks require Leanplum SDK version iOS 1.4.1 / Android 1.3.1 or later.

At Leanplum, we believe every piece of content should be tested. That's why we make it easy to A/B test campaigns and see how successful they are at driving user behavior and ROI.

By withholding your campaign from a group of users, you can measure how successful the campaign is at driving behavior compared to the holdback group. Using the holdback feature in conjunction with Conversion events can give an even clearer picture of the user behavior generated by the campaign.

Add a holdback group

To add a holdback group, click in the Holdback input in the lower-left corner of the Audience step and type in a new holdback percentage.

Users who are assigned to the holdback do not receive the campaign — no need to set up a separate A/B test. We handle all of that behind the scenes! Leanplum creates a control group that doesn't receive the campaign and compare results to the test group that does.

📘

Users who don't receive your campaign as a result of a holdback will track a "held back" event in Analytics and their user profile. This event means the user would have got the campaign if no holdback was set.

You can also use the holdback to send a campaign to a smaller portion of users. For example, to send your campaign to just 20% of your audience, move the control group's distribution (the holdback group) up to 80%.

Analyze holdback results

Leanplum tracks a few Events for users targeted in a campaign with a Holdback. Everyone who qualifies for the campaign's audience and delivery settings will track a Holdback impression event in Analytics and in their user profiles, whether or not they receive the campaign.

The users who receive the campaign will also track a message Sent event in Analytics and in their user profiles. The users who do not receive the campaign (because of the holdback) will trigger a Held back event at the time the message/action in the campaign would have been sent.

All these events are visible in the user profiles in the Users tab, as well as in the Analytics for the message itself. See Events tracked by default for a full list with descriptions of Leanplum's default events.

📘

• For one-time and recurring campaigns, the Holdback test automatically finishes two days after the last message occurrence.
• For triggered campaigns, the test automatically finishes 30 days after the campaign is finished.

❗️

Holdbacks can't be added to previously published campaigns. To add a Holdback to an existing campaign, please create a new copy of your campaign.

We recommend using Holdback in conjunction with Conversion events to help you understand the full impact that your campaigns are having on users.