Regression testing of a mobile app on ten devices is a headache without a cloud device farm. Different manufacturers, OS versions, screen resolutions — every test has to be run locally or on real devices in the office. AWS Device Farm solves this by using the AWS ecosystem: the same IAM roles, CLI, access policies. Our experience shows that 80% of problems at first launch are due to improper test packaging or device pool configuration. This article will help you avoid these mistakes. We have configured Device Farm for 30+ projects, and the average reduction in regression testing time was 70% — allowing clients to stop buying physical devices and focus on code quality.
How to Set Up Testing on AWS Device Farm?
Why AWS Device Farm Instead of BrowserStack or Sauce Labs?
If your project is already on AWS (CodePipeline, CodeBuild, S3), Device Farm integrates seamlessly: no separate login, billing through AWS, centralized IAM policies. For companies with data residency requirements, this is a plus — devices are in the same regions as the rest of the infrastructure. Device Farm supports APK/IPA and Instrumentation tests (Espresso) without additional layers. Compared to BrowserStack, Device Farm gives more control over the device pool, although the interface is less intuitive.
What Is Included in the Setup?
| Component | Description |
|---|---|
| Project and IAM roles | Project creation, minimal permissions |
| Device pool | Filter by platform, OS version, manufacturer |
| Test packaging | Examples for Appium, Espresso, XCUITest |
| CI/CD integration | CodePipeline, GitHub Actions, GitLab CI |
| Reports and notifications | Slack, Telegram, S3 |
| Documentation | Instructions for your team, scripts |
Types of Tests: Comparison
| Type | Speed | Support | Packaging Complexity |
|---|---|---|---|
| Appium | Medium | Android, iOS | Medium (dependencies in ZIP) |
| Espresso | High | Android (4.3+) | Low (standard APK) |
| XCUITest | High | iOS (9.0+) | Medium (xcodebuild) |
How to Package Appium Tests for Device Farm?
Packaging Appium tests is a common cause of failures. Device Farm requires a ZIP with package.json, node_modules, and tests. Structure for WebdriverIO:
tests.zip/ ├── package.json ├── package-lock.json ├── node_modules/ └── test/ └── specs/ └── login.test.js WebdriverIO config:
// wdio.conf.devicefarm.js exports.config = { runner: 'local', specs: ['./test/specs/**/*.js'], capabilities: [{ platformName: process.env.DEVICEFARM_DEVICE_PLATFORM_NAME, 'appium:deviceName': process.env.DEVICEFARM_DEVICE_NAME, 'appium:platformVersion': process.env.DEVICEFARM_DEVICE_OS_VERSION, 'appium:app': process.env.DEVICEFARM_APP_PATH, 'appium:automationName': process.env.DEVICEFARM_DEVICE_PLATFORM_NAME === 'iOS' ? 'XCUITest' : 'UiAutomator2', }], hostname: 'localhost', port: 4723, }; Device Farm starts the Appium server automatically; variables are passed automatically.
Native Tests: Espresso and XCUITest
Espresso runs faster than Appium because it executes in the same process as the app. Upload APK and test APK via AWS CLI:
PROJECT_ARN=$(aws devicefarm create-project --name "MyApp" --query 'project.arn' --output text) APP_UPLOAD=$(aws devicefarm create-upload --project-arn $PROJECT_ARN --name "app-debug.apk" --type ANDROID_APP --query 'upload.{arn:arn,url:url}' --output json) APP_URL=$(echo $APP_UPLOAD | jq -r '.url') APP_ARN=$(echo $APP_UPLOAD | jq -r '.arn') curl -T app/build/outputs/apk/debug/app-debug.apk "$APP_URL" TEST_UPLOAD=$(aws devicefarm create-upload --project-arn $PROJECT_ARN --name "app-debug-androidTest.apk" --type INSTRUMENTATION_TEST_PACKAGE --query 'upload.{arn:arn,url:url}' --output json) TEST_ARN=$(echo $TEST_UPLOAD | jq -r '.arn') curl -T app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk "$(echo $TEST_UPLOAD | jq -r '.url')" aws devicefarm schedule-run --project-arn $PROJECT_ARN --app-arn $APP_ARN --device-pool-arn $POOL_ARN --name "Espresso Run" --test type=INSTRUMENTATION,testPackageArn=$TEST_ARN,filter="com.example.LoginTest" For iOS, use XCUITest; package via xcodebuild.
Step-by-Step Guide to Setting Up Device Farm
- Create a Device Farm project and set up IAM roles with minimal permissions (devicefarm:* on the project).
- Build a device pool: select platform, OS versions, manufacturers, and filter by AVAILABILITY: HIGHLY_AVAILABLE to avoid queues.
- Upload the app (APK/IPA) and test package via AWS CLI or console. Ensure tests are properly packaged (ZIP with dependencies).
- Start a test run, selecting the pool and test type (Appium, Espresso, XCUITest).
- Configure CI/CD integration (CodePipeline, GitHub Actions) and notifications (Slack, Telegram).
Integration with CI/CD
Add a step in buildspec.yml for CodePipeline:
phases: build: commands: - ./gradlew assembleDebug assembleAndroidTest - APP_ARN=$(aws devicefarm create-upload --project-arn $DEVICE_FARM_PROJECT_ARN --name "app.apk" --type ANDROID_APP --query 'upload.arn' --output text) # ... upload and run - aws devicefarm get-run --arn $RUN_ARN --query 'run.result' The CodeBuild IAM role must have devicefarm:* permissions on the project.
Why Do Tests Fail on the First Run?
Common errors:
- Incorrect Appium test packaging — ZIP without node_modules.
- Device pool without AVAILABILITY: HIGHLY_AVAILABLE filter — run queues for an hour.
- Wrong environment variables — Device Farm passes DEVICEFARM_DEVICE_PLATFORM_NAME and others.
- Lack of write permissions to S3 for artifacts.
Official AWS documentation recommends using highly available pools to minimize wait times.
Analyzing Results
Artifacts are accessible via API:
aws devicefarm list-artifacts --arn $JOB_ARN --type FILE --query 'artifacts[*].{name:name,url:url}' --output table Typical artifacts: Logcat, Screenshots, Video, Test spec output. Our engineers set up automatic report delivery to Slack or Telegram.
Case Study: How Device Farm Setup Saved 40 Hours per Month
One client — a fintech startup with an Android app tested on 20 devices locally. Each regression run took 8 hours of manual work. After setting up Device Farm with GitLab CI integration, the time dropped to 1.5 hours fully automated. We configured a pool of 10 parallel devices, connected Slack notifications, and exported reports to S3. The cost of the cloud device farm turned out lower than maintaining an in-house device fleet.
Our engineers hold AWS certifications and have 10+ years of experience in mobile development. We guarantee your first run will succeed within 3 days of starting work.
Contact us for a free project assessment. Order AWS Device Farm setup and get your first successful run in 3 days. All scripts and documentation remain with you.
For more details on Device Farm, see the official AWS documentation.