ui-automation: Automated UI Code Generation from Sketch and Images

ui-automation is a tool built by the Xianyu team that can automatically generate maintainable UI code from Sketch designs or PNG images, achieving quality comparable to senior front‑end engineers.

Background: Improving UI development efficiency has progressed from hand‑written code to XML and visual editors. With advances in computer‑vision and deep‑learning, the team believes it is possible to surpass manual coding.

Problem: Existing approaches (traditional scanning, pix2code) can render identical UI but suffer from poor maintainability and limited dynamic capabilities. ui‑automation focuses on solving code‑quality issues.

Workflow: Information extraction → DSL derivation → Target platform code (e.g., Flutter, Weex).

Information extraction layer: Extracts full but noisy data from Sketch, or clean data from images.

DSL layer: Transforms flat upstream information into a hierarchical tree with complete layout constraints.

Template layer: Generates platform‑specific code from the DSL.

Basic UI elements: Shape, Text, Image, each consisting of content, render style, and flexbox‑based layout style.

DSL processing: Consists of a grouping layer and a sorting layer.

Grouping layer uses binary splitting based on parent‑child or sibling relationships, employing overlap and projection models to build a binary tree.

Group(...children: View[]){}

type slice = (views: View[]) => View[][] // sliceByOverlaps(views) || sliceByProjection(views)

After grouping, a flattening step merges same‑direction parent‑child nodes to reduce tree depth.

const flattenOnDirection = (view: View, parentDir: FlexDirection) => { ... }

Sorting layer orders nodes left‑to‑right and top‑to‑bottom based on container direction.

Additional steps include auxiliary line detection, density splitting, grid scanning, middle‑line perception, and merging background layers.

Attribute inference layer: Derives container direction, flow vs. absolute positioning, size (using a greedy model), special layouts (grid, left‑right alignment), main‑axis and cross‑axis alignment, position, and padding.

ui‑automation is already deployed in many Xianyu business scenarios and continues to evolve.

For recruitment, contact: [email protected]