Selective Testing to Accelerate CI/CD for Android UI Tests at Agoda

Agoda engineers observed that developers were frequently blocked by lengthy CI pipelines, especially when waiting for UI and integration tests to validate changes. Running a monolithic suite of nearly 900 UI tests for their Android app caused instability, long delivery times (over 2.5 days), and a poor developer experience.

To address these issues, they adopted a selective testing approach. The workflow starts with developers writing code, running only the tests they believe are affected, and pushing changes. Determining which tests are impacted requires mapping each test to the code it executes, which they achieve using code‑coverage tools such as JaCoCo.

They enabled UI‑test coverage in Gradle:

android {
buildTypes {
debug {
testCoverageEnabled = true
}
}
}

When UI tests run, the Emma coverage tool records bytecode execution into an .exec file on the device. After each test batch, the file is pulled from the device and stored in the module’s build directory.

Using JaCoCo, they parse each .exec file to extract covered lines:

/*** Extract code lines from coverage file */
fun parseToLines(codeCoverageFile: File): List
{
val loader = ExecFileLoader().also { it.load(codeCoverageFile) }
return loader.executionDataStore.contents.map { it.name }
}

The extracted data is transformed into a mapping of code lines to the tests that executed them:

// CoverageData: mapping of each line to the list of tests that covered it
[
com/packageName/ClassName$Method1 : [test1],
com/packageName/ClassName$Method2 : [test1, test2]
]

With this coverage data, a test selector can compare PR changes against the mapping and run only the tests that touch the modified code, dramatically reducing the number of tests executed per PR.

To make coverage data available without regenerating it for every PR, they generate it once on the main branch, store it in the cloud, and let CI jobs download it. The selector then picks the relevant tests based on this cached data.

They also tackled flaky tests by only uploading coverage for successful tests; failed tests are treated as new and always run, ensuring that flaky failures are not silently ignored.

Results show a significant reduction in test execution time and CI failures, confirming that selective testing is an effective optimization for CI/CD pipelines. The team plans to extend this approach to other repositories.

In summary, selective testing driven by fine‑grained code‑coverage data, combined with a device farm and caching strategy, improves CI efficiency, maintains test quality, and accelerates development cycles for Android applications.