How to Extend Flutter UI Automation Beyond Native Tools: A Practical Guide

Background

Flutter pages cannot be directly located by native testing tools, which creates many difficulties for automation. Although Google provides Flutter driver and integration_test, they have several drawbacks: they do not work well with hybrid‑stack apps, element locating is weak, and they depend on VMService, requiring debug or profile builds.

Flutter driver analysis

Initial attempts using Appium showed that many UI elements were merged into a single region, forcing reliance on fragile XPath selectors. Flutter driver works only for pure Flutter apps and its source was examined. The framework connects to VMService, serializes commands as JSON over a WebSocket, and executes them on the app side.

Example of the JSON command used to retrieve a widget’s text:

{
  "jsonrpc":"2.0",
  "id":5,
  "method":"ext.flutter.driver",
  "params":{
    "finderType":"ByValueKey",
    "keyValueString":"counter",
    "keyValueType":"String",
    "command":"get_text",
    "isolateId":"isolates/4374098363448227"
  }
}

Based on this protocol, a Python wrapper was built to drive Flutter tests using uiautomator2 and facebook‑wda, enabling testing of hybrid apps.

from flutter_driver.finder import FlutterFinder
from flutter_driver.flutter_driver import FlutterDriver
import uiautomator2 as u2

if __name__ == "__main__":
    d = u2.connect()
    driver = FlutterDriver(d)

if pageFlutter is True:  # Use Flutter driver for Flutter pages
    driver.connect("com.it592.flutter_app")
    finder = FlutterFinder.by_value_key("input")
    driver.tap(finder)
    time.sleep(1)
    print(driver.getText(FlutterFinder.by_value_key("counter")))
else:
    d(text="increase").click()

However, Flutter driver still has limitations: it cannot batch‑operate on multiple elements, many developers do not assign keys, there is no official inspect tool, and the project is no longer maintained.

integration_test analysis

Google replaced Flutter driver with integration_test, which runs on devices or emulators without extra drivers. It allows using any Flutter API, can be packaged as an APK/IPA for execution on Firebase Test Lab, and supports isolated page‑level tests. Nevertheless, its element locating mechanism is the same as Flutter driver, so the same weaknesses persist, plus additional issues such as the need to rebuild the app for script changes, extra waiting functions for data loading, poor suitability for full‑stack end‑to‑end tests, and limited extensibility.

Xianyu UI automation solution

After evaluating the official tools, Xianyu chose to extend native automation frameworks with image‑processing techniques. The architecture combines native element locating (using name, label, or stable XPath) with OCR for text‑based elements and image matching for pure graphics. A trained classifier identifies common controls such as product cards, prices, icons, and avatars.

To improve maintainability, page elements are encapsulated in separate classes, decoupling them from test logic. This layered approach has been applied to Xianyu’s performance automation, running over 500 successful executions with a success rate above 98%.

Key components of the solution are illustrated in the diagrams below.

Conclusion

Both Flutter driver and integration_test lack mature support for hybrid‑stack applications. Flutter driver can be extended for pure Flutter apps, while integration_test remains less stable for mixed environments. Leveraging native automation with OCR and image‑processing offers a lower‑cost, higher‑benefit alternative for comprehensive UI testing.