Extending Flutter UI Automation: Analysis of Flutter Driver, Integration Test, and Xianyu's Hybrid Approach

Flutter pages cannot be directly located with native testing tools, which hinders UI automation. Although Google provides Flutter Driver and Integration Test, they have several drawbacks for hybrid stack apps: limited element locating, dependence on VMService (requiring profile or debug builds), and lack of support for mixed native‑Flutter screens.

To overcome these issues, the team chose to extend native testing tools for Flutter UI testing. The article analyzes the principles of Flutter Driver and Integration Test and presents Xianyu's practical solution.

Flutter Driver

Initial attempts with Appium merged many elements into a single region, forcing XPath usage, which is fragile. Flutter Driver works only for pure Flutter apps and provides element locating via VMService. The communication protocol is JSON over WebSocket, e.g.:

{
    "jsonrpc":"2.0",
    "id":5,
    "method":"ext.flutter.driver",
    "params":{
        "finderType":"ByValueKey",
        "keyValueString":"counter",
        "keyValueType":"String",
        "command":"get_text",
        "isolateId":"isolates/4374098363448227"
    }
}

Based on this protocol, a Python wrapper was built using flutter_driver and uiautomator2:

from flutter_driver.finder import FlutterFinder
from flutter_driver.flutter_driver import FlutterDriver
import uiautomator2 as u2

if __name__ == "__main__":
    d = u2.connect()
    driver = FlutterDriver(d)
    if pageFlutter is True:  # flutter page
        driver.connect("com.it592.flutter_app")
        finder = FlutterFinder.by_value_key("input")
        driver.tap(finder)
        time.sleep(1)
        print(driver.getText(FlutterFinder.by_value_key("counter")))
    else:
        d(text="increase").click()

Limitations of Flutter Driver include inability to batch‑operate elements, reliance on developers adding keys, no built‑in inspect tool, and the project being discontinued.

Integration Test

Integration Test replaces Flutter Driver but still relies on flutter_test for element operations. Its advantages are:

Test scripts can use any Flutter API.

Tests can run on devices/emulators (e.g., Firebase Test Lab) without extra drivers.

Each page test is independent, enabling page‑level testing.

However, it inherits the same locating problems and adds new constraints: scripts must be repackaged after each change, extra waiting logic is needed for data loading, it is not friendly for full‑flow tests, and extensibility remains weak.

Xianyu UI Automation Solution

After evaluating official frameworks, Xianyu chose to augment native automation with image‑processing techniques. The architecture combines native locators (name, label, stable XPath) with OCR for text‑based elements and image matching for pure graphics. A training set classifies common controls (cards, prices, icons, avatars) using image classification.

Scripts are layered: page objects encapsulate element definitions, separated from test logic, improving maintainability. The solution has been used in performance automation, running over 500 executions with a success rate above 98%.

Conclusion

Both Flutter Driver and Integration Test lack mature support for hybrid apps. Flutter Driver can be extended, but the newer Integration Test still suffers from the same limitations. For hybrid scenarios, augmenting native automation with OCR and image‑matching offers lower cost and higher ROI.

Acknowledgements: thanks to SLM and TMQ for underlying support. References: Flutter Driver (https://flutter.dev/docs/cookbook/testing/integration/introduction) and Integration Test (https://flutter.dev/docs/testing/integration-tests).