Ke-FUT: A Self‑Developed Flutter UI Automation Framework for Mobile Testing

1 Introduction Flutter is Google’s cross‑platform UI framework that enables a single codebase to run on multiple devices with near‑native performance. While many Chinese internet companies have adopted Flutter, the increase in Flutter pages has raised the cost of manual regression testing because native automation tools cannot recognize Flutter elements.

To address this, the team at 贝壳 (Beike) investigated existing solutions and found that Flutter‑driver, Appium‑flutter‑driver, and Airtest each have drawbacks such as invasive code changes, lack of Python support, or high integration cost. Consequently, they created their own framework called Ke‑FUT (Ke‑FlutterUiTest).

2 Solution Research The three surveyed tools were evaluated on stability, business integration cost, and compatibility, and none satisfied the requirements, prompting the development of Ke‑FUT.

3 Technical Principles and Implementation

3.1 Overview Ke‑FUT follows a two‑step process similar to native UI automation: (1) obtain element IDs and (2) drive view elements. The architecture consists of three layers – Application, Bridge, and Service – mirroring the UIAutomator2 model used for Android.

3.2 Architecture The Service layer runs inside the Flutter app, exposing element analysis and view‑driving capabilities via VMService. The Bridge layer forwards messages from the Application layer to the Service, and the Application layer provides a test‑engineer‑friendly API.

3.3 Obtaining IDs Ke‑FUT uses Flutter’s VMService and InspectorService. By connecting to VMService, sending a show command puts the app into SelectedMode, allowing the InspectorService to receive widget information. The following Dart snippet shows how the connection is established:

InspectorService inspectorService;
Future<void> main(List<String> args) async {
  final String url = args[0];
  final uri = normalizeVmServiceUri(url);
  FrameworkCore.init(url);
  final connected = await FrameworkCore.initVmService('', explicitUri: uri, errorReporter: (message, error) {});
  if (connected) {
    inspectorService = await InspectorService.create(serviceManager.service).catchError((e) {}, test: (e) => e is FlutterInspectorLibraryNotFound);
  } else {
    safe_exit(1);
  }
  await inspectorService.invokeServiceMethodDaemonNoGroup('show', {'enabled': true});
}

When an element is selected, the _getSelectedWidget method of WidgetInspectorService returns a JSON payload. Using the AspectD AOP framework, the method is hooked to inject an autoId field generated by IdGenerator.idGenerator(current):

@Execute("package:flutter/src/widgets/widget_inspector.dart",
        "_WidgetInspectorService", "-_getSelectedWidget")
@pragma("vm:entry-point")
Map<String, Object> _getSelectedWidget(PointCut pointcut) {
  print('call _getSelectedWidget');
  final Element current = WidgetInspectorService.instance.selection?.currentElement;
  Map<String, Object> map = pointcut.proceed();
  if (current != null) {
    map['autoId'] = IdGenerator.idGenerator(current);
  }
  return map;
}

3.4 ID Generator The generator maps a Flutter Element to a stable ID by traversing the element tree, discarding irrelevant nodes, and handling multi‑child render objects. A simplified version of the generator is shown below:

class IdGenerator {
  static void _parse(List<Element> allList) {
    for (int i = 0; i < allList.length; i++) {
      var desc = allList[i].toStringShort();
      switch (desc) {
        case 'WidgetInspector':
          i++;
          break;
        case 'Icon':
          idList.add(desc);
          break;
        case 'Stack':
          _mutilChildren2Id(allList.sublist(i), desc);
          break;
        default:
          break;
      }
      element2Id[allList[i]] = idList?.join('/') ?? '';
    }
  }

  static void _mutilChildren2Id(List<Element> chanList, String type) {
    MultiChildRenderObjectElement multiChildRenderObjectElement = chanList[0];
    int i = 0;
    int finalIndex = 0;
    multiChildRenderObjectElement.visitChildren((element) {
      if (element == chanList[1]) {
        finalIndex = i;
      }
      i++;
    });
    idList.add('$type[$finalIndex]');
  }
}

3.5 Driving Views After obtaining an element ID, test scripts use the FUTClient to send commands to FUTService via WebSocket. FUTService runs inside the Flutter app, exposing APIs such as getPositionById, setText, and assertText. The following Dart function demonstrates how to compute an element’s screen coordinates:

Map id2Position(String id) {
  initElement2IdMap();
  if (element2IdMap.containsKey(id)) {
    Element element = element2IdMap[id];
    RenderObject renderObject = element.renderObject;
    if (renderObject is RenderBox) {
      Offset offset = renderObject.localToGlobal(Offset(0, 0));
      Size size = renderObject.size;
      double x = (offset.dx + size.width / 2) * window.devicePixelRatio;
      double y = (offset.dy + size.height / 2) * window.devicePixelRatio;
      return {'x': x, 'y': y};
    }
  }
  return {'x': 0, 'y': 0};
}

The Python‑based FUTClient implements these APIs. Example snippets include a click_id helper and a get_position_by_id request:

def click_id(self, element_id, logtext):
    """Click an element by its ID"""
    if element_id:
        position_map = flutter_client.get_position_by_id(element_id)
        print("ID-{}的坐标值为:{}".format(logtext, position_map))
        if isinstance(position_map, dict) and position_map.get('x') != 0:
            x = position_map['x']
            y = position_map['y']
            self.d.click(x, y)
            logger.info("点击元素:{}".format(logtext))
            time.sleep(2)
        elif position_map.get('x') == 0 and position_map.get('y') == 0:
            logger.info("ID:{}返回坐标值为:{}, 元素不存在!".format(element_id, position_map))
            raise AssertionError("ID:{}返回坐标值为:{}, 元素不存在!".format(element_id, position_map))
        else:
            raise AssertionError("元素异常:{}".format(logtext))

def get_position_by_id(self, id: str) -> dict:
    id_en = id.encode('utf-8')
    base64_id = base64.b64encode(id_en)
    tem_map = {'method': 'getPositionById', 'id': base64_id}
    json_map = simplejson.dumps(tem_map)
    self.client.send(json_map.encode('utf-8'))
    while True:
        data = self.client.recv(1024)
        if not data:
            break
        position_map = simplejson.loads(data.decode('utf-8'))
        return position_map
    return None

4 Practice The framework has been deployed in the 贝壳租赁 (Beike Rental) business, covering dozens of pages. Test engineers can obtain element IDs via flutter attach and the internal bk_weditor tool, then write test cases such as the following price‑adjustment script:

def test_puzu_change_price(self, init):
    self.flutter_base.click_text("调价")
    self.flutter_base.click_id(elements.price_icon, "调价")
    self.flutter_base.set_text(elements.price_icon, "322")
    self.flutter_base.click_text("保存")
    self.base.assert_toast("调价成功")

5 Future Outlook Ke‑FUT is now integrated into the KeMTC automation platform for both Android and iOS. While it already supports common UI actions, the project is still in its early stage. Future work includes fuzzy matching for IDs and text, broader scenario coverage, and inviting mobile testing engineers to contribute feedback.