Automating IS Regression Testing with SSIM Image Comparison and Async Rendering

1. IS Overview

IS (Inspiration Spaces) is a consumer‑facing marketing product from the Coohom team that lets users select sample rooms, view them in panoramic mode, and replace content assets in a WYSIWYG manner, covering custom, finished, and hard‑fit industries.

2. Background

IS depends on many upstream services (customization, hard‑fit, rendering, Coohom, etc.). Changes in any upstream code can break IS functionality, yet downstream teams only provide APIs and lack deep knowledge of IS, leading to heavy reliance on Coohom testing for regression and high testing pressure.

3.1 IS Principle

The core of IS is to feed a base image and layered image data into a compositor, then display the result via a panoramic module.

The compositor uses a new technique: base image + layered image, where the layered image consists of a product image and a depth map indicating the product’s position on the base.

The compositor combines the base and selected product layers into a single‑face panoramic texture, which is handed to the panoramic module for display.

3.2 Regression Principle

Because IS assembles a panoramic view from base and layered images in a specific order determined by depth maps, each base image, product layer, and depth map must be correct. A product layer comprises six faces (u, d, l, r, f, b); all six must match the expected reference.

To verify consistency, the system compares images using the Structural Similarity Index (SSIM), which focuses on luminance, contrast, and structure, reflecting human perception of edges and textures. SSIM scores close to 1 indicate high similarity.

SSIM formula (simplified):

Since rendering may introduce noise, a threshold (e.g., 0.9) is set instead of requiring a perfect score.

3.3 Regression Solution

The automated regression workflow includes:

Select sample rooms and replacement items (covering 17 styles, each with at least two replacement products).

Submit rendering jobs to the backend (asynchronous rendering).

Poll rendering status every 30 seconds, abort after 10 hours.

When rendering finishes, invoke a lightweight web service (built with Bottle) that fetches original and new layer URLs, computes SSIM scores, and returns similarity metrics.

This reduces manual clicks from over 119 per sample room to a single automated run.

4. Regression System Architecture

5. Sequence Diagram

6. Result Display

The UI shows a list of pre‑defined sample rooms with rendering status and comparison results, and a detailed view of per‑layer SSIM scores. Anomalies (e.g., blurred handles) are highlighted for further investigation.

7. Summary

7.1 Benefits

Downstream teams can run regression without learning IS internals.

Detects rendering anomalies caused by upstream changes, having already uncovered five production issues.

Saves at least two hours per release by automating test case execution.

7.2 Issues & Improvements

Image comparison is time‑consuming; multithreading could speed it up.

Small differences on tiny layers cause low scores; need tolerance thresholds.

Uniform thresholds across resolutions affect precision; per‑resolution thresholds and SDK APIs for direct layer comparison are desired.