Comprehensive UI Automation Framework for the Dali Smart Homework Lamp: Challenges, Solutions, and Future Directions

The Dali Smart Homework Lamp integrates a dual‑camera design that enables remote homework assistance for parents, combining a smart desk lamp with a vision system to recognize and explain tasks, synchronizing learning status between the lamp and the parent app.

Product features compare the smart hardware (including voice, gesture, image, and voiceprint inputs) with traditional mobile apps, highlighting differences in test object, input methods, defect‑recall cost, and performance concerns such as memory, CPU, and storage.

Key testing difficulties include rapid business iteration causing case maintenance pressure, multimodal inputs (voice, image capture), noisy‑free environment requirements, technology updates (WebView, Flutter, Lynx) that prevent native element access, and the need for coordinated testing across multiple apps and devices.

Current benefits stem from the Shoots automation framework, which the team extends to achieve richer test capabilities, lower case‑writing cost, reduced consumption cost, cross‑version compatibility, and the ability to capture "hard‑to‑test" data for bug reporting.

Automated Script Generation

def auto(self):
    self.wait(5)
    self.step("开始复现脚本，请尽快重命名")  # todo 需要重命名
    if not self.checkEnterWin('主界面'): return False
    self.touchOcrKey('作业批改')
    if not self.checkEnterWin('拍照_主界面'): return False
    self.tapKey('单题搜')
    self.tapKey('take_photo')  # text = 单题搜
    if not self.checkEnterWin('拍照框题页'): return False
    self.tapKey('cropper_iv_retake')  # text = 整页检查
    if not self.checkEnterWin('拍照_主界面'): return False
    self.tapKey('iv_flashlight')  # text = 补光
    self.tapKey('iv_flashlight')  # text = 补光
    self.tapKey('iv_close')
    if not self.checkEnterWin('主界面'): return False
    return True

Custom Pop‑Up Handling

POP_WINDOWS = {
    '学生端_坐姿检测_结果': ['btn_start_finish', 1, 'KEYCODE_BACK', 0.5],
    '学生端_作业模式_作业提醒': ['iv_close', 1],
    '学生端_准备升级': ['update_reboot_later', 1],
    '学生端_作业开始提示': ['iv_close', 1],
    '勋章_获得': [[320, 1300], 1],
}

def dealWithPopWindows(self, win):
    popWindows = self.getPopWindows()
    if popWindows is None or len(popWindows) == 0:
        return win
    for k, v in popWindows.items():
        if window_manager().isWindow(win, k):
            for item in v:
                if type(item) == int or type(item) == float:
                    self.wait(item)
                elif type(item) == list and len(item) == 2:
                    self.mainDevice().touch(item)
                elif item.startswith('KEYCODE'):
                    Log.warning("发现Pop窗口[{}],尝试通过按Key [{}]关闭".format(k, item))
                    self.mainDevice().keyEvent(item)
                else:
                    Log.warning("发现Pop窗口[{}],尝试通过按键 [{}]关闭".format(k, item))
                    win.clickItem(item)
                    self.wait(1)
            # Recursive handling for complex scenarios
            return self.dealWithPopWindows(self.refreshWin())
    return win

The framework promotes modular case design by extracting atomic operation groups, assigning meaningful names, and encapsulating them to reduce maintenance overhead when requirements change.

Pop‑up windows are centrally defined in constants, allowing test scripts to ignore their handling details; the framework automatically processes them based on the defined actions.

Rich testing methods include automatic UI element verification by analyzing the Android view tree, image/text search with fuzzy matching, repeated attempts for unstable page loads, and multi‑device coordinated actions.

Audio and video mocking capabilities involve using auxiliary devices or injecting audio files to simulate real‑world scenarios, reducing environmental dependencies.

Case updates support automatic AB‑experiment compatibility by configuring page flow mappings, enabling a single script to handle multiple version branches and automatically switch based on detected UI states.

Result consumption generates difference reports between current and expected page snapshots, matches pages against historical snapshots, and auto‑generates error analyses and bug tickets, thereby reducing manual effort.

Log collection is configurable via keyword‑to‑file mappings, facilitating targeted log extraction for issue diagnosis and strict‑mode fault collection.

Automatic diagnosis runs both as plugins during test execution and post‑run based on logs, feeding into the error analysis pipeline.

Metrics quantify the ROI of UI automation by measuring test result consumption time, script maintenance cost, and overall efficiency, providing data to correct misconceptions about low automation ROI.

Test left‑shift practices such as branch inspection trigger automated builds and tests on every commit, allowing early detection of quality regressions.

Performance data collection during testing monitors memory leaks and threshold breaches, while algorithm acceptance leverages the same mocking infrastructure for end‑to‑end validation.

Future outlook emphasizes two directions: fine‑grained UI validation through sampling, screenshots, layout memory, and human assertions; and intelligent traversal with automatic assertions to autonomously discover and flag UI issues for human review.