Pause, copy, or delete your test

Pausing an A/B test

You may want to pause a test to reconsider your strategy or to prevent users from seeing a variant.

What happens when you pause a test?

When you pause a test, all of the users in the test will be served the control group's experience on their next app start. Once you restart your test, the users will resume seeing their variant group experience when they start the app.

🚧

If a message you are testing is live, pausing your test may cause all targeted users to start receiving the message. To prevent this, make sure your message is also paused before you pause a test. This is because the status of messages in your test (live, paused, etc.) are independent of the test's status, so if you pause the A/B test but leave the messages active, messages will still go out to all users that match the message's targeting criteria (even those in a test group that had the message disabled). See more on A/B testing messages.

Also note that pausing a test can affect the integrity of your results. Because different types of users interact with your app at different times.

For example, let's say you were testing a new messaging strategy to maximize the number of returning users. Then, you pause your test for one day and restart it. This seems like a minor change, but what if you paused on a Saturday, when more users tend to use your app? Or, what if you paused the test early in the morning, before some users go to work? Different types of users interact with your app at different times, and pausing can skew your test results more than anticipated.

How do I keep new users from entering a test without affecting the experience of my current users?

To keep new users from entering a test without affecting your users' current test groups, edit the target of the test to exclude virtually all of your users (e.g. set the target to only one specific userID). If your test is sticky (magnet button activated), all the users who were previously in the test will remain in their same variant groups, but no new users will be able to join.

This is different from pausing because rather than serving everyone the control experience, all the users in your test will remain in their original variants.

Copying an A/B test

To make a new test based on one of your existing tests, you can make a copy. Click the ( ... ) button in the upper right corner of the screen, then select Copy from the dropdown that appears.

You can copy tests within your app, or you can copy a test from one app into another one of your organization's apps.

This will create a new version of the test with all of your old test's Targets, Goals, Distribution, and variants pre-loaded into the settings. You can alter these settings and the test's title before sending the new test out to your users.

🚧

Always double check all settings and variants before starting any A/B test — especially when using copies. If an old test had a message with scheduled delivery, for example, the dates may no longer be correct. Or, if you copy a variable test from one app, but that variable doesn't exist in the other app, your test won't work correctly.

Also, always be aware of the end result of the original test you are copying. If you already rolled out a variant or message to your users, starting a new test based on the old copy may conflict with a change you already made.

Deleting an A/B test

Click the ( ... ) button in the upper right corner of the test, then click Delete.

If you delete a test that is currently live, all users will return to the default experience (control) on their next app start. To avoid this, it's better to finish your test and select a winning variant before deleting the test.

Abort send

Sometimes mistakes happen, and you may want to stop users from receiving a message variant that already went live. There's no way to unsend a message that has already been delivered, but "Abort send" can stop some messages that haven't been delivered yet.

Click the ( ... ) button in the upper corner of a completed test to reveal Abort send.