A/B testing a message

When A/B testing a message, there are a few additional settings to consider when setting up a test.

Create the test

To create a test for a message, either add the test directly from the message composer or add your message to a new A/B test using the Add dropdown in the test table.

Audience settings

Most A/B test settings, like goals and impression criteria, work with messages just like any other test, however, your test and message audience settings can both affect which users will receive your message. Pay close attention to the following settings when setting up a test with a message.

Targets

It's generally best to make sure the targets for your test and your message are the same — this prevents users outside of the test from receiving the message. Creating the test from the message composer automatically adds your message targets to the A/B test composer, so they will already match.

If the target audience for your message is larger than the target for your A/B test, the dashboard will display a warning to alert you that users outside of your test will likely receive your message.

For example, if you target "All Users" on the message, but restrict the A/B test target to US users only, all users outside the US will still receive the message, but only US users will enter the A/B test.

🚧

Pausing test messages

We recommend pausing any messages you are A/B testing. This allows the A/B test's target to control who receives the message, (preventing the message target from being evaluated and sending outside the test). This is especially important when using the audience slider. See below for more detail.

Audience slider

The audience slider determines what percentage of your Target audience will be in your A/B test. You can use it to send the test to a smaller portion of your audience, or to slowly roll out a test to your full audience.

If you use the audience slider, Leanplum will prompt you to pause your message before publishing your test. This is because if your message is live, the part of your message’s audience that is held out of the test can still receive your message. Just follow the prompt or return to your message to make sure it is paused before you proceed.

📘

Immediate send messages do not work with the audience slider.

To send an immediate message to a smaller portion of your users, consider using User Buckets in your target (and don't forget to make sure your test and message targets match!). User buckets will target a random portion of your users.

The message Holdback feature can also make it easier to test out a new message on a portion of your audience.

❗️

Once you start a test, you can only increase your audience size with the slider. You will not be able to make the portion of users receiving your test smaller after the test has started.

Disable a message for test groups

When setting up an A/B test for your message, you may want to test the effects of not sending the message (the control) against actually sending it (the variant).

To disable the message for the control group, uncheck the Enabled setting underneath the control group, and enable for in the variant.