Low-Cost Automated App Response Time Testing Using External Camera and Image Comparison

Mobile app performance testing involves many metrics, but response time most directly impacts user experience; existing methods for measuring it have limitations in cost and accuracy.

Known methods

Method 1: On Android, use AM to calculate app launch time or DisplayManager logs for activity opening time.

Method 2: Insert logging code into the program to timestamp specific events.

Method 3: Record the screen with a camera or take screenshots and compute response time via manual or programmatic image comparison.

Drawbacks of existing methods

Method 1 often yields results that differ from human perception because system logs cannot capture asynchronous network interactions and cannot measure fine‑grained UI rendering.

Method 2 requires invasive code changes, increasing maintenance cost and is rarely used in production.

Method 3 either needs high‑speed cameras, which are expensive, or relies on device screenshots that interfere with the test environment and can introduce errors due to resource consumption or visual changes such as time stamps.

Key focus

Method 3 best matches the goal of visual response‑time measurement, but its drawbacks motivate a solution that combines automation, external image capture, and an optimized image‑comparison algorithm.

Key point 1: No pollution

Goal: Capture the screen without contaminating the test environment.

Solution: Use an external camera to record the device screen.

Implementation: 1) Windows PC with an external camera; 2) Connect the phone and camera to the PC, display the camera feed in a window; 3) Use Windows API to capture window images.

Explanation: Capture frequency directly affects accuracy; images are kept in memory during a test case and written to disk only once at the end, achieving a 20 ms per screenshot rate.

Key point 2: Noise resistance

Goal: Image comparison should ignore color differences and local changes, approximating human visual perception.

Solution: Use a perceptual‑hash algorithm that focuses on overall image structure and disregards fine details.

Implementation roadmap

Technical implementation

1. Automation driver: MonkeyRunner.

2. Image comparison: Python PIL implementation of perceptual‑hash.

3. Screenshot: Windows API with 20 ms interval.

4. Time consistency: All timestamps use the Windows system clock.

The test case assumes that after an operation the UI reaches a stable state; the start time is recorded when the action is triggered, and the end time is determined by scanning the image sequence backward to find the last frame that changed.

Example pseudo‑code for measuring app launch time:

1. Launch app and record start time

2. Start screenshot program to record screen

3. Wait sufficiently long for the app to load

4. End case, save captured screenshots

5. Use perceptual‑hash algorithm to compare images and compute end time

Pros and cons

Advantages

1) Test environment remains untouched; the PC handling screenshots does not affect the app.

2) High precision: 20 ms capture interval yields theoretical error around 50 ms.

3) Effective noise filtering for small dynamic UI elements.

4) Portable: works on Android and can be adapted to iOS.

5) Low cost and easy to use; only basic hardware (regular camera) and automation skills are required.

Disadvantages

1) Not CI‑friendly due to reliance on an external camera.

2) Requires image preprocessing for large dynamic screen areas (masking, cropping).

3) Screenshot interval (20 ms) does not match typical 60 Hz (16.7 ms) screen refresh, causing slight timing error; a possible remedy is using HDMI capture cards via MHL/Slimport.

4) High memory usage on the PC to store frequent screenshots before writing to disk.

Author

Yin Fei, Senior Test Engineer at Baidu Nuomi, with extensive experience in mobile testing tools and quality assurance for client products.