Better A/B Testing with Firebase
This post originally appeared on The Firebase Blog.
Announcing Better A/B Testing with Firebase
Getting to Know the New A/B Testing Feature
With the new A/B testing feature, you can create an A/B test that will allow you to play with any combination of values that you can control through Remote Config. Setting up an A/B test allows you to define how the experiment will behave in a number of different ways, including determining how many of your users are involved with the experiment at first…
…how many variants you want to run, and how your app might behave differently for each variant…
…and what the goal of the experiment is.
Different experiments might have different desired goals, and A/B testing supports a number of common outcomes, like increasing overall revenue or retention in your app, reducing the number of crashes, or increasing the occurrence of any event you’re measuring in Google Analytics for Firebase, such as finishing your in-app tutorial.
Once you’ve defined your A/B test, Firebase takes over by delivering these different variations of your app to randomly-selected members of your audience. Firebase will then measure your users’ behavior over time, and let you know when an experiment appears to be performing better, based on those goals you’ve defined earlier. Firebase A/B testing measures these results for you with the same Bayesian statistical models that power Google Optimize, Google’s free testing and personalization product for websites.
Using A/B Tests for Better Onboarding: A Case Study
Fabulous, a motivational app for building better habits, recently made improvements to their app’s onboarding by using Firebase A/B testing. When the user first starts an app, Fabulous shows them how to complete a habit, presents them with a letter about forming better habits, and then asks them to commit to a simple routine. The team suspected that if they removed a few steps from this onboarding process, more people might complete it.
Some of the screens a typical user encounters when first using Fabulous.
So they ran an A/B test where some users didn’t see the letter, others didn’t see the request to commit to a simple routine, and others skipped both of those steps. The Fabulous team found that by removing both of these steps from the onboarding process, there was a 7% improvement in the rate of users completing the onboarding flow. More importantly, they confirmed that this shorter onboarding experience didn’t have any impact on their app’s retention.
Test Your Notifications, Too!
You also have the ability to A/B test your app notification messages through the Firebase Notifications console. You can try out different versions of your notification message and see which ones lead to more users opening up your app from that notification, or which messages lead to users performing some intended goal within your app, like making a purchase.
Getting Started
A/B testing is available in beta to all Firebase developers starting today. If you’re excited to get started, you should make sure that your app is wired up to use Remote Config and/or Firebase Cloud Messaging, and that you’ve updated these libraries to the latest and greatest versions. You can always find out more about A/B testing in our documentation, or check out the A/B Test Like a Pro video series we’ve been building.
Then, head on over to the Firebase Console and start making your app better — one experiment at a time!
