Using impression criteria

Impression criteria allow you to set when a user is tracked for the analytics of an A/B test. This is an optional setting when setting up an A/B test.

If you want to limit your A/B test to just users that have completed a certain Event (e.g. made a purchase, or viewed the signup page), or reached a certain State in your app (a level in a game), impression criteria can help yield more focused results.

For example, say you’re testing something on the settings section of your app. When a user opens the app and enters the test, they may never see what’s being tested on the settings page, but they would still count in your test's metrics/analytics if they fit your target group. By setting an impression event like “Settings Page Selected”, you would ensure that only users who fire that event will impact the test results/analytics.

📘

If you set impression criteria for a test, you will not be able to see the users who don't reach the impression event in your A/B test analytics.

The Users in your test's target who don't reach the impression criteria will still be assigned to your test, but you just won't see them in analytics. If this isn't what you want, it's completely ok to not set any impression criteria for your experiment.

When to use impression criteria

🚧

Use caution

Impression criteria cannot be removed from an A/B test without skewing your results. If you aren't sure if you need one, it's probably safer to create a test without any impression criteria.

Generally, the choice to use impression criteria boils down to two things:

1. How certain you are of the Events/States that should trigger analytics tracking
2. How deep in your app are you testing

Setting impression criteria may be a good idea if you are testing a feature that's unlikely to be encountered shortly after a user starts the app — or if you're only concerned with users that have reached a certain State or Event (e.g. Add to Cart).

For example, if a gaming company is testing a change in the difficulty of Level 10 in their game, they may want to set an impression criteria of "User enters Level 10" since many users will not reach that point in the game, and the test should focus on just tracking those users.

However, if you're testing something on the home page, or aren't quite sure what should trigger the tracking, we recommend skipping impression criteria. You can always use some filters in Analytics to narrow your results down after the fact.

For example, if you are testing a new user onboarding flow, it may not be necessary to specify any impression criteria, as all new users who open the app will be exposed to the test almost immediately.

How Leanplum evaluates qualified users with impression criteria

Users are still evaluated against your test targets and assigned to any relevant A/B tests on app start. Users who enter your test just won’t show up in analytics until they trigger the impression criteria (if one is specified).

How it works:

1. User opens the app
2. Leanplum is initialized
3. Test targets are evaluated
4. If user meets the targeting criteria, they are randomly placed in the control or the variant
5. User triggers the event that has been chosen as the impression criteria
6. User is tracked in the analytics at that point

📘

If an impression criteria is added to a test after the test has started, it will not retroactively remove un-impressed users from analytics. The analytics will still include all users who were added to the test before adding the impression criteria. Going forward, all users who reach the impression criteria will show in the test's analytics.