CodeGenie: Generate HarmonyOS UI Code from Images in 2 Minutes

From Hand-Coding to Image-to-Code

HarmonyOS AI-assisted programming tool CodeGenie introduces a new feature: upload an app screenshot or design mockup (JPG/PNG) and automatically generate compilable, runnable HarmonyOS ArkTS page code, including ArkUI components, layout structure, and component properties, rapidly completing page scaffolding.

This is not simple image recognition but intelligent generation that truly understands page structure. After uploading an image, AI performs multi-level analysis and processing:

Basic Element Recognition

Component type classification: Distinguishes different UI component types such as buttons, input fields, list items, etc.

Image region segmentation: Identifies images, icons, background images, and analyzes their dimensions and proportions.

Layout Structure Parsing

Container relationship inference: Infers layout container types (Column, Row, Grid, etc.) from spatial distribution of elements.

Spacing and size measurement: Calculates element dimensions and mutual spacing relationships, generating corresponding layout parameters.

Alignment and distribution: Recognizes left/right/top/bottom alignment, equal spacing, space-between, etc., and outputs matching configuration parameters.

Style and Text Extraction

Style attribute parsing: Identifies visual properties like background color, font, border, and generates corresponding style code.

Text content extraction: Extracts textual information from the interface — titles, body text, button labels — preserving original style and formatting as much as possible.

Why "Image Alone Can Reproduce"

Industry Challenges in Screenshot-to-Code

"Image-to-code" (Screenshot2code) has seen early research with small models (pix2code, img2cook) and current large-model approaches (Flame-Code-VLM, Cursor, V0). Two main challenges remain:

Insufficient visual understanding precision: UI pages cover component types, sizes, styles, layouts, and relative positions. Unlike UX drafts that contain detailed style specs, image styles are pixel-based and require additional parsing. Low image understanding accuracy leads to significant visual deviations between generated results and original designs.

Unstable code effect quality: Evaluated on two dimensions:

Compilable preview rate: For coarse-grained engineering code generation, if generated code cannot be previewed instantly, it greatly impacts user experience, especially in 0-to-1 prototyping. Current VLM/LLM single-pass generation has low compilable preview rates, often requiring manual debugging (10+ minutes) with no guarantee. In HarmonyOS scenarios, models are more prone to hallucination.

Code quality: Merely restoring visual layout and style is insufficient for production; code must follow software engineering norms (e.g., proper foreach loops, avoiding variable redundancy). Current LLMs still show large gaps in understanding and implementing development frameworks and coding standards in complex scenarios, limiting practical usability.

Our Solution: Parse-Then-Generate Architecture

Given the low usability of end-to-end VLM/LLM image-to-code generation, we adopted a "parse first, generate later" architecture. First, high-precision visual parsing using VLM and object detection models achieves 85%+ element parsing accuracy. Then, enhanced with HarmonyOS knowledge base, three types of agents — requirement analysis, code generation, verification and repair — collaborate to produce final ArkTS code. Currently achieves page code generation in ~2 minutes, compilable preview rate near 100%, and ~40% online adoption rate since beta.

1. High-Precision Element Parsing with Large and Small Models

Direct VLM-based object detection faces issues: output token overflow, high inference latency, and unstable fine-grained parsing due to label system mismatch. Balancing fine-tuning cost and latency, we use a small model for initial block detection, then VLM for detailed parsing.

Module division: A dedicated UI element detection small model performs initial region recognition, outputting location and category information for each module. These regions feed into subsequent fine-grained parsing; part of the label system maps directly to HarmonyOS container components.

VLM detailed parsing: With the first-layer modules, VLM parses each module concurrently, extracting component names, dimensions, text, fonts, positions, etc., building a complete component tree. Predefined common fonts and layout info from the HarmonyOS knowledge base are incorporated to reduce generation noise and model hallucination.

Parsing result example (JSON):

{
  "global_info": {
    "app_type": "xx...",
    "page_function": "首页/个人中心/...",
    "color_scheme": ["主色HEX", "辅色HEX"]
  },
  "layout_structure": [
    {
      "area_type": "顶部导航栏",
      "components": [
        {
          "component_type": "返回按钮",
          "position": [["5%","3%"], ["15%","10%"]],
          "style": {
            "icon": "箭头方向(left)",
            "color": "#333333",
            "size": "24x24pt"
          },
          "interaction": {
            "type": "clickable",
            "target": "返回上一页"
          }
        }
      ]
    }
  ],
  "detail_annotations": {
    "key_elements": ["搜索框", "主功能入口"],
    "dynamic_area": "轮播图(第3屏)",
    "business_flow": "旅游酒店预订流程入口"
  }
}

2. Hybrid Generation Architecture: LLM + DSL2Code

Despite improved LLM ArkTS generation, coarse-grained engineering code still suffers from TypeScript syntax confusion and version compatibility issues. We built a dual-path mechanism fusing LLM generation with DSL-to-code conversion.

HarmonyOS-knowledge-enhanced LLM generation path: After obtaining the component tree, a requirement analysis agent clarifies page layout and functional design, then LLM generates code. HarmonyOS component documentation and sample code serve as a RAG knowledge base to enhance ArkTS generation. A verification-repair loop iterates using compilation results and error messages to ensure runnable output.

DSL2Code deterministic generation path: To address LLM uncertainty and latency, for well-defined scenarios we implemented a DSL2Code system that directly converts the parsed component tree into code. DSL2Code is efficient and deterministic but may lack engineering standards; a refactoring agent automatically restructures and standardizes output to improve maintainability.

One-Minute Workflow Walkthrough

Using "HarmonyOS World APP Homepage" as an example, the complete flow from screenshot to HarmonyOS page code:

Launch generation command: In CodeGenie input box, type "/" to open command panel, select "Generate Page" to enter page generation UI.

Upload page screenshot: Click add image, upload the homepage screenshot, click send.

AI automatic parsing and generation: The tool automatically:

Parses components and layout: search bar, top background image, main content area, article list, bottom navigation bar, etc.

Restores text content: titles, descriptions, statistics, category icons.

Extracts style information: colors, font sizes, spacing, visual attributes.

Generates ArkTS code and real-time preview rendering, ensuring visual consistency.

Save and integrate into project: Click "Save to Project" to automatically generate:

Complete ArkTS code files

Associated resource files

Directly compilable and runnable project

All generated code files are saved in the generated directory by default, ready to compile and run, effectively reducing debugging cost.

The current version is deeply optimized for five key vertical domains — food, travel, shopping, news, and education — with relatively precise generation results. Other domains also have certain generalization capability; continuous quality and scope improvements are ongoing.

Conclusion: A New Development Paradigm Arrives

This is more than an efficiency tool; it is an intelligent bridge connecting visual design and code implementation. It frees developers from tedious UI coding, allowing focus on business logic and user experience innovation. Whether you are an entrepreneur needing rapid prototype validation or a developer seeking higher efficiency, this tool opens new possibilities. Try generating your first HarmonyOS page with a single image.