Classify Frontend Components with Pipcook’s Image Classification Pipeline

Background

In many frontend projects you may have screenshots of UI components and need an automatic way to identify whether a given image is a button, navigation bar, table, or chart. This is a classic image‑classification task that can power code generation, automated testing, and performance monitoring.

Scenario Example

For a backend‑admin form generator you might need to distinguish line charts, bar charts, pie charts, and ring charts. After training, the model predicts a probability array for each class, e.g. [[0.1, 0.9, 0.05, 0.05]] where the highest confidence (0.9) indicates a line chart.

The label map that translates numeric indices to class names looks like:

{
  "column": 0,
  "line": 1,
  "pie": 2,
  "ring": 3
}

Data Preparation

Organize your dataset into train, validation, and test folders, each containing subfolders named after the classes (e.g., line, ring, column, pie) with the corresponding images inside.

Download the prepared dataset from the provided link.

Start Training

Use Pipcook with the following pipeline configuration:

{
  "plugins": {
    "dataCollect": {
      "package": "@pipcook/plugins-image-classification-data-collect",
      "params": { "url": "http://ai-sample.oss-cn-hangzhou.aliyuncs.com/pipcook/datasets/component-recognition-image-classification/component-recognition-classification.zip" }
    },
    "dataAccess": { "package": "@pipcook/plugins-pascalvoc-data-access" },
    "dataProcess": { "package": "@pipcook/plugins-image-data-process", "params": { "resize": [224, 224] } },
    "modelDefine": { "package": "@pipcook/plugins-tensorflow-mobilenet-model-define", "params": { "batchSize": 8, "freeze": false } },
    "modelTrain": { "package": "@pipcook/plugins-image-classification-tensorflow-model-train", "params": { "epochs": 15 } },
    "modelEvaluate": { "package": "@pipcook/plugins-image-classification-tensorflow-model-evaluate" }
  }
}

The pipeline uses the following plugins:

@pipcook/plugins-image-classification-data-collect – downloads the dataset.

@pipcook/plugins-pascalvoc-data-access – converts the dataset to Pipcook format.

@pipcook/plugins-image-data-process – resizes images to a uniform size.

@pipcook/plugins-tensorflow-mobilenet-model-define – defines a lightweight MobileNet model (use ResNet for more complex data).

@pipcook/plugins-image-classification-tensorflow-model-train – trains the model.

@pipcook/plugins-image-classification-tensorflow-model-evaluate – evaluates the model on the test set.

Training typically converges within 10–20 epochs. Example log:

Epoch 1/15 ... loss: 0.0604 - accuracy: 0.9823 - val_loss: 8.8755 - val_accuracy: 0.4112
Epoch 2/15 ... loss: 0.0056 - accuracy: 0.9993 - val_loss: 5.5883 - val_accuracy: 0.4925
... 
Epoch 15/15 ... loss: 5.1657e-05 - accuracy: 1.0000 - val_loss: 1.9073e-08 - val_accuracy: 1.0000

After training, an output npm package is generated. Install dependencies:

cd output
BOA_TUNA=1 npm install

Run predictions with:

const predict = require('./output');
(async () => {
  const result = await predict('./test.jpg');
  console.log(result); // e.g., [[0.1, 0.9, 0.05, 0.05]]
})();

The model returns a probability for each class; you can post‑process these probabilities as needed.

Summary

By following this pipeline you can train an image‑classification model to recognize frontend UI components, integrate it into your development workflow, and extend the approach to your own datasets. Future articles will cover multi‑component detection in design drafts.