How AI Powers Automatic Frontend Code Generation

Background

As one of the four technical directions of Alibaba's Frontend Committee, the Frontend Intelligent project proved its value during the 2019 Double‑11 event, with 79.34% of the new modules' code generated automatically.

Layer Overview

The focus is on the basic component recognition layer, which identifies predefined UI components in images to help downstream systems optimize layer structures and semantics.

Overall Solution

The pipeline follows a classic algorithm‑engineering flow: sample acquisition → sample evaluation → model training → model evaluation → model prediction .

Sample Acquisition & Evaluation

Samples are generated by encoding UI libraries to ensure balanced and diverse component data, following principles such as equal class distribution, inclusion of special scenarios (e.g., text overlay), and padding around wireframe components.

After generation, samples are validated for labeling errors and class distribution statistics.

Model Selection

Based on recent PASCAL VOC benchmarks, the top‑ranked one‑stage detector YOLO was chosen for transfer learning to quickly prototype the component detector.

YOLO works by resizing images to 416×416, learning classification features via a CNN, and applying non‑maximum suppression during inference.

Model Evaluation

Mean Average Precision (mAP) on a COCO‑style test set is used to measure accuracy, revealing lower performance on small targets such as text elements.

Model Prediction Processing

During inference, images are resized to 416×416, and post‑processing with OpenCV refines bounding boxes via gradient edge detection, significantly improving IoU scores.

Conclusion

The Frontend Intelligent team currently supports over 20 basic component types and plans to further refine sample granularity, standardize data management, and open the dataset for external use.