Large-Scale UI Sample Generation for Alibaba 99 Promotion Module Recognition

Background: In the previous article "Detailed Implementation of Alibaba 99 Promotion Page Content Recognition", the algorithm model used for automatic testing was introduced.

Difficulty: The promotion contains hundreds of modules and thousands of pages. Accurate module identification requires tens of thousands of samples per module, which is infeasible to label manually.

Idea: Generate massive UI samples by describing each module’s View with a DSL and combining it with dynamic ViewModel data, then rendering the DSL into images.

DSL Description: A module is represented as a JSON‑like DSL that lists atomic elements (Text, Image, Shape) and their hierarchical groups, similar to an HTML DOM tree. Each element includes its frame (x, y, width, height), type, value (for text/image), and a minimal set of style properties.

{
  "layers": [{
    "frame": {"y":354,"x":44,"height":32,"width":312},
    "id":2,
    "type":"text",
    "value":"Adidas Stan Smith",
    "textStyles":{"fontFamily":"Helvetica, sans-serif","fontSize":24}
  }, ...]
}

Obtaining the DSL: three possible ways – (1) extract from the code repository, (2) generate from Sketch design files (using the internal tool imgcook), (3) extract from the rendered browser page. The third method was chosen because it can faithfully reproduce the final UI without dealing with diverse code patterns.

Technical Process (using Puppeteer):

Hide default CSS properties – discard properties that match browser defaults.

Remove ineffective properties – keep only the ~20 properties actually used by developers.

Transform handling – convert the matrix() transform returned by window.getComputedStyle into screen‑relative translate values.

css {
  alignSelf: 'auto',
  ...
}

# Python snippet for transform conversion
if 'transform' in style and 'matrix' in style['transform']:
    matrix = style['transform'][7:-1].split(',')
    translate = list(map(float, matrix[-2:]))
    translateResult = list(map(str, [d * screenshotShape[0] / 750 for d in translate]))
    matrix[-2:] = translateResult

Rendering DSL to Images: Map DSL element types to HTML tags, recursively render groups, and capture screenshots with Puppeteer.

ViewModel Dynamic Data: The same View can be reused across different promotion events (99 promotion, Double‑11, etc.) by swapping dynamic data such as product images and titles, producing millions of variant samples.

Result: The pipeline generates tens of thousands of high‑quality samples per module, boosting model accuracy to over 98%.

Future Outlook: Automate DSL adjustments (e.g., corner radius variations, noise injection) by letting the model decide how to diversify samples.