Creating an A/A test & interpreting results

An A/A test is a test where the variant group is the same as the control group.

What is the purpose of an A/A test?

It may seem silly to test two identical groups, but the goal of an A/A test isn't to test a change to one of the groups — the goal is to evaluate the statistical rigor of the tool used to run experiments.

How to build an A/A test

You can set up an A/A test in Leanplum by creating an A/B test, then building an identical Control and Variant(s).

What if your A/A test groups show statistically significant differences?

No two audience groups will be exactly alike. Even with good experiment design (sufficiently large sample sizes, random selection, and user assignment), the metrics for each test group will likely show some differences.

If many metrics show significant differences, it's possible there's a statistical problem with the tool's experimentation methodology. If you are experiencing these types of results in an A/A test, contact your CSM.