When we researched mobile marketing solutions that could handle our daily events and successfully manage a [large] user base… Leanplum was the only platform that fit the bill.
As a Leanplum customer, App Annie’s B2C Apps team recently ran a series of mobile A/B tests to optimize onboarding conversion rates for its “My Data Manager” app. The app is a solution that helps its own users track mobile data usage and save money.
The statistically significant result led to a 15 percent increase in a conversion rate that would directly impact App Annie’s business.
We sat down with Eric MacKinnon, App Annie’s Director of Product Management for B2C Apps, to discuss key learnings from their mobile A/B tests and their experience with Leanplum.
As a large, enterprise brand, what were your experiences running mobile A/B tests?
App Annie:
“If you’re running an app that has millions and millions of users, product changes carry a lot of risk. People often think of mobile A/B testing from the perspective of startups trying to increase conversion rates. The focus tends to be on validating the presence of positive impact. But over time, as your app grows in size, you start worrying more about the flip side — ensuring that necessary changes (such as updating and modernizing the app) don’t have an adverse effect on user-behavior. Of course, I certainly want improvements, but what’s more important is that we avoid the risk introducing changes that hurt the business.”
How has Leanplum improved your ability to make educated decisions?
App Annie:
“Leanplum has been absolutely instrumental in helping us improve how we onboard new users, while at the same time keeping our current users happy.
Through Leanplum, we carefully manage tests for a large existing user base and are able to measure against a wide variety of different metrics. This is crucial, as product changes often have unintended consequences. Having a holistic view of the impact to user behavior is the only real way to ensure you’re making the right call.”
How do you evaluate a successful test in Leanplum?
App Annie:
“Evaluating tests in Leanplum is quite seamless, even when we’re looking at several different metrics at one time. Normally, considering potential trade-offs between multiple metrics to determine overall impact can be tricky. But with Leanplum, all of the metrics are automatically bubbled up. We’ve found this helps to get an overall sense of how a change may negatively or positively impact user behavior. It’s really easy to analyze our results at a glance and gain insights for future testing.
As for setting up a test, Leanplum makes it super easy. We have specific goals in mind when we introduce a product change, so we typically start by selecting the most appropriate metric(s) to run the test against. (All other metrics are available to us later so it’s not crucial to get this right at the outset.) We then segment which kinds of users we want included in the test. In some cases, this might be only new users in a specific country; in others, existing users who have set up a VPN connection.”
Sample Metrics
“Leanplum gives us the option to test multiple variants against one another, as well as control what percentage of users are presented each variant. Once we’re set up with our variants, the platform gives us an estimate of how long we’ll likely have to run the test before having enough of a sample size to reach statistical significance.
Our tests take anywhere from a single day to a couple of weeks to run. Speed and flexibility are key for our team, and Leanplum is fast enough for us to quickly gather actionable data to make incremental improvements in our next iteration.”
What is the best part of working with Leanplum?
App Annie:
“Leanplum’s customer support is there for me around the clock and around the globe. We have teams stationed in the US, Canada, the Netherlands, and China. Leanplum is always available to chat about questions or strategies, no matter the time zone.
Leanplum provides actionable feedback that we can implement into the product. It’s not just about A/B testing. Leanplum offers best practices and strategies tailored specifically for our app. Our experience is always personalized, and we trust Leanplum’s expertise to move our brand forward.”
