Suppose you launched an A/B test for a new onboarding, and a week later conversion dropped — and it's unclear whether it's random or an effect. We've seen many times how incorrect Remote Config activation breaks the experiment: the user sees both variants in one session, making data useless. To get reliable results, you need to control several technical aspects. Our implementation experience — over 5 years, dozens of projects — ensures that experiments yield statistically significant conclusions.
Why Firebase A/B Testing needs preparation
Firebase A/B Testing is an overlay on Remote Config. You create an experiment in the console: define a parameter, a control group (current value) and variants (new ones). Firebase distributes users on the server, and with fetchAndActivate each gets their value. But if activation happens after screen render — the user sees a switch, and the experiment gets contaminated.
A typical mistake is overlapping experiments: two tests change the same screen independently. Firebase allows parallel experiments, but the responsibility for avoiding conflicts lies with the team. We maintain a table of active experiments with parameters being changed.
How to avoid experiment overlap?
Overlap is a common cause of unreliable data. Solution: before launching a new experiment, check that no active test changes the same Remote Config parameter. Use a single table (e.g., in Confluence) with columns: experiment name, changed parameters, start date, expected end date. We guarantee that after auditing your active experiments, conflicts will be eliminated.
Why statistical significance is critical
Without sufficient data volume, the experiment result is just noise. Firebase uses Bayesian statistics: it computes the probability that the variant is better than control. But stopping the test early can give a high probability by chance. For a 5% conversion rate, you need at least 500 conversions per group. With smaller sample sizes, decisions lead to metric degradation. Our engineers always calculate the required sample size before launch.
How we set up experiments
- Formulate the hypothesis: what we change, which metric we impact, what effect we expect.
- Create a parameter in Remote Config with type and default value.
- Verify that client code reads the parameter before rendering the target screen.
- Launch the experiment in Firebase console with the target Analytics event.
- Monitor statistical significance: for conversions below 5%, we need at least 500–1000 conversions per group.
- When sufficient data is reached, decide: lock the variant or revert to control.
Implementation on iOS
// Config already set up via RemoteConfig // In the experiment: parameter "paywall_position" = "bottom" (control) / "center" (variant) remoteConfig.fetchAndActivate { [weak self] _, _ in let position = RemoteConfig.remoteConfig()["paywall_position"].stringValue DispatchQueue.main.async { self?.paywallViewModel.position = position == "center" ? .center : .bottom } } It's mandatory to log the trigger event — Firebase A/B Testing uses it to mark "experiment seen":
Analytics.logEvent("experiment_paywall_viewed", parameters: [ "variant": RemoteConfig.remoteConfig()["paywall_position"].stringValue ?? "unknown" ]) Implementation on Android (Kotlin + Jetpack Compose)
val remoteConfig = FirebaseRemoteConfig.getInstance() remoteConfig.fetchAndActivate().addOnCompleteListener { task -> if (task.isSuccessful) { val position = remoteConfig.getString("paywall_position") // Apply to UI viewModel.position = if (position == "center") Position.Center else Position.Bottom } } On Flutter (via firebase_remote_config)
final remoteConfig = FirebaseRemoteConfig.instance; await remoteConfig.setConfigSettings(RemoteConfigSettings( fetchTimeout: const Duration(seconds: 10), minimumFetchInterval: const Duration(hours: 1), )); await remoteConfig.fetchAndActivate(); final paywallPosition = remoteConfig.getString('paywall_position'); Activation strategy comparison
| Strategy | When to use | Risk |
|---|---|---|
| fetchAndActivate immediately | Screen loads after activation | None if config applied before UI |
| fetch + activate later | On cold start to avoid lag | User may see default |
| Only on initialization | For parameters unchanged in session | Slow response to changes |
What's included in our work
- Setting up Remote Config with parameters specific to the experiment.
- Typed access to experimental parameters (to eliminate typos).
- Integration at the trigger point (before rendering the target screen).
- Configuring target events in Firebase Analytics for conversion measurement.
- Consultation on experiment design: hypothesis, metric, minimum sample size.
Timeline and cost
From 1 day (if Remote Config is already integrated) to 3 days (from scratch, including analytics and consultation). Cost is calculated individually — we estimate the workload after reviewing your project. Request an audit of your experiment and get free consultation.
Example experiment metrics
| Parameter | Control group | Variant group |
|---|---|---|
| CTA button position | Bottom | Center |
| Conversion to sign-up | 12.3% | 14.7% |
| Achieved significance | – | 87% (insufficient) |
| Recommendation | – | Continue test |
Compared to a custom solution: Firebase A/B Testing is 3x simpler — no need to write server-side distribution logic, and Bayesian statistics are built in. Our experience — over 5 years in mobile development — guarantees correct setup. Contact us for consultation — we'll help you avoid typical mistakes and get reliable results. Order Firebase A/B Testing integration for your app.