Configuring Real Device Testing with Firebase Test Lab

Configuring Real Device Testing with Firebase Test Lab

Emulators lie. Not maliciously, but systematically: GPU rendering differs, Bluetooth is unavailable, the camera is simulated, memory behavior is not like a real device. At TrueTech, with over 5 years of experience on mobile projects, we've encountered this dozens of times. Firebase Test Lab provides access to physical devices — Pixel 8, Samsung Galaxy S24, tablets, old Xiaomi units with custom firmware. A test that passes on an emulator can crash on Samsung One UI due to a custom WebView or modified permissions system. Order a turnkey setup — we'll prepare the matrix, Robo Script, and CI integration in 2–3 days.

Why Emulators Fail and Real Devices Don't

Emulators don't reproduce hardware specifics: different memory behavior, lack of real sensors (accelerometer, gyroscope), emulated camera. Real devices expose issues with push notifications (APNs/FCM), background execution (Background fetch), and deep links (Universal Links / App Links). According to our data, about 30% of crashes found on physical devices are not reproducible on an emulator. Using Firebase Test Lab ensures your app is ready for real users.

What Firebase Test Lab Can Do

Two testing modes:

Instrumentation Tests — runs your Espresso / XCUITest tests on selected devices. Full control over what is checked.

Robo Test — automatic crawler. Launch an APK without writing a single test; Robo navigates the UI via the accessibility tree, taps buttons, fills fields with random data, and looks for crashes. Useful for quick checks of new builds before release.

Supported platforms: Android (Espresso, UI Automator, Appium) and iOS (XCUITest). Firebase Test Lab for iOS has a smaller device pool, but real iPhones and iPads are still more valuable than a simulator.

Setup: From Build Upload to Results

Android

Build APK and test APK:

./gradlew assembleDebug assembleAndroidTest

Upload and run via gcloud:

gcloud firebase test android run \
  --type instrumentation \
  --app app/build/outputs/apk/debug/app-debug.apk \
  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
  --device model=Pixel8,version=34,locale=ru,orientation=portrait \
  --device model=a54xnsxx,version=13,locale=ru,orientation=portrait \
  --results-bucket gs://my-project-test-results \
  --results-dir "run_$(date +%Y%m%d_%H%M%S)"

model=a54xnsxx is a Samsung Galaxy A54. List available models: gcloud firebase test android models list.

Device Matrix

Choosing a matrix is a separate analytical task. No point in running tests on 50 devices. Principle:

A realistic matrix for most projects: 3–5 devices. More is costlier and slower, but not necessarily more informative.

iOS

For iOS, you need an .ipa with a development signature (not distribution). Upload:

gcloud firebase test ios run \
  --test MyAppTests.zip \
  --device model=iphone15pro,version=17.4,locale=ru_RU,orientation=portrait

MyAppTests.zip is an archive with the .xctestrun file and test products. Build via xcodebuild build-for-testing.

Robo Test with Custom Script

The plain Robo crawler sometimes gets stuck on the login screen — it doesn't know credentials. Robo Script allows you to define initial actions:

[
  {
    "eventType": "VIEW_TEXT_CHANGED",
    "replacementText": "[email protected]",
    "elementDescriptors": [{"resourceName": "com.example.app:id/email_input"}]
  },
  {
    "eventType": "VIEW_CLICKED",
    "elementDescriptors": [{"resourceName": "com.example.app:id/login_button"}]
  }
]

After authentication, Robo continues crawling the authenticated part of the app. This is a quick way to check regressions without writing tests.

CI Integration

GitHub Actions:

- name: Set up gcloud
  uses: google-github-actions/setup-gcloud@v2
  with:
    service_account_key: ${{ secrets.GCLOUD_SERVICE_ACCOUNT_KEY }}
    project_id: my-firebase-project

- name: Run tests on Firebase Test Lab
  run: |
    gcloud firebase test android run \
      --app app-debug.apk \
      --test app-debug-androidTest.apk \
      --device model=Pixel8,version=34 \
      --timeout 10m

A service account with the Firebase Test Lab Admin role is needed. Results are automatically saved to Cloud Storage — screenshots, videos, logcats, XML reports.

Analyzing Results

In the Firebase Test Lab console, for each run you get:

• Video recording of the test
• Logcat with tag filters
• Screenshots at key points
• Coverage report (if jacoco enabled)
• XML results in JUnit format (for Allure / Jenkins)

What's Included in Turnkey Setup

We offer a full cycle:

• Analysis of your device park based on analytics data
• Configuration of a 3–5 device matrix
• Writing a Robo Script for authentication (if needed)
• CI integration (GitHub Actions, GitLab CI, Bitrise)
• Team training on working with results
• Guarantee of correct operation for 5 days after delivery

Our engineers have experience with 30+ projects using Firebase Test Lab. Contact us for a project assessment — we'll find the optimal configuration.

Timeline

2–3 days — setting up Firebase Test Lab, configuring the device matrix, CI integration, writing Robo Script for authentication. Plus time for initial analysis of results and fixing device-specific issues. Cost is calculated individually.

How Firebase Test Lab Saves Time on Regressions

Robo tests without a single line of code find crashes and ANRs on real devices faster than manual testing. According to our data, this saves up to 40% of QA cycle time. Combined with instrumentation tests, you get full coverage in a matter of hours.

Mobile app testing automation: from unit to E2E

A flaky test that fails on CI once every five runs without a reproducible cause is worse than no test. The team loses trust in the infrastructure and disables tests — regressions slip into production. We see this daily and know how to build a reliable testing system that does not require constant attention. Contact us for a free consultation and test architecture assessment.

Why are flaky tests dangerous?

One unstable check can break the pipeline, blocking a release. Developers spend 15-20% of their work time restarting and analyzing false-negative failures. Automation without stability is not saving efficiency but losing it. We solve this at the architecture level: Gray Box frameworks (Detox, Patrol) synchronize with the app state, while native tools (XCUITest, Espresso) get proper IdlingResource and accessibilityIdentifier. Result: stability >99% on CI.

What should you unit test in mobile apps?

On iOS XCTest is the foundation. Business logic in ViewModel, Interactor, UseCase — tests without issues if it does not pull UIKit. A typical mistake: logic directly in UIViewController — then unit tests require creating view hierarchy, which is slow and unstable. The solution is to move logic to services with @testable import.

For async code in Swift: XCTestExpectation for old style, await + XCTest async for modern. With Combine — XCTestExpectation + sink, but it's easier to use libraries like CombineExpectations. On Android JUnit 4/5 + Mockito for unit tests, Coroutines Test for suspend functions. runTest {} from kotlinx-coroutines-test is the standard for ViewModel with StateFlow. Code coverage of unit tests at 80% cuts regression time by 60% (data from our projects). Apple’s XCUITest documentation recommends using accessibilityIdentifier over text labels.

UI Tests: Stability Over Coverage

XCUITest (iOS) and Espresso (Android) — native UI tests. They run fast, are integrated with IDE, but test one platform. The main issue with XCUITest is fragile selectors. app.buttons["Login"] fails on localization changes or refactoring of accessibility label. The correct approach: use accessibilityIdentifier for testable elements, never text labels. Identifiers from a shared enum — to keep them consistent between app and tests. Experience shows: this practice reduces flakiness by 90%.

Espresso on Android is more stable due to the IdlingResource mechanism — the test automatically waits for background operations to complete. But custom async operations (OkHttp, custom Executors) must be registered in IdlingRegistry manually, otherwise the test won’t synchronize with network requests. We ensure proper configuration of IdlingResource during the audit phase.

Detox and Patrol: End-to-End for React Native and Flutter

Detox — E2E framework for React Native, developed by Wix. Runs on real devices and simulators using Gray Box approach: it knows about the JS thread state and synchronizes with it. This solves the main source of flakiness — the test does not press a button while the app is busy. Detox setup is non-trivial. Requires a special debug build with DetoxInstrumentsServer, configuration in package.json, and no separate Appium server. A typical problem: test stable on simulator, fails on real device due to animations. Solution: animations: disabled in Detox config for E2E build.

Patrol — analog for Flutter. Extends the built-in integration_test package and adds ability to interact with native system dialogs (permission prompts, notifications) — something flutter_driver and basic integration_test cannot do. For CI, use via patrol test --target integration_test/app_test.dart. Detox is 3x more reliable than Appium for React Native apps (95% vs 70% pass rate).

Appium: Cross-Platform at a Cost

Appium — when you need to cover iOS and Android with the same tests. Uses WebDriver protocol on top of XCUITest and UiAutomator2 drivers. Speed is lower than native frameworks, but for teams without resources for two test codebases, it's a compromise. Appium 2.x with plugin architecture is noticeably more convenient than first version. appium-doctor diagnoses the environment — useful when setting up CI.

CI and Parallelization

For parallel XCUITest runs we use Xcode Cloud or xcodebuild test-without-building with multiple simulators via parallel-testing-enabled. Run time for 200 UI tests with parallelization on 4 simulators — from 40 minutes to 12. On Android we use Firebase Test Lab with sharding.

How We Do It: Process

1. Audit current code and CI — evaluate flakiness, coverage, bottlenecks. We typically find 15-20% of tests are flaky.
2. Design test architecture — choose framework, selectors, mocks.
3. Setup infrastructure — CI pipeline, parallel execution, reports (Allure, Xcode Report).
4. Write tests — unit, UI, E2E, performance (XCTMetrics, Macrobenchmark).
5. Integration and stabilization — run 200+ tests, catch flaky cases. Past projects show flakiness drops from 15% to 2%.
6. Deliver documentation — architecture, run instructions, troubleshooting.

Deliverables

• Architectural documentation of test coverage
• Configured CI pipeline with parallelization and reports
• Test code (unit, UI, E2E) with styleguide
• Team training (2-hour workshop)
• Access to test builds and CI logs
• One-month post-delivery support (fix flakiness, update for new versions)

Estimated Timelines

Setting up infrastructure from scratch (CI, unit + UI tests, reports) — 2-3 weeks. Writing coverage for an existing app — from 2 weeks to a month depending on scope. We will assess your project in 2 days — contact us. Get a customized automation plan for your project – reach out today. 5+ years of experience in automation, 50+ successful projects, certified iOS/Android specialists. We guarantee test stability >98% on CI after implementation.