Build a Pure Front‑End Icon Recognition Tool with Machine Learning

Background

Front‑end component libraries often rely on Iconfont, but as icon sets grow, naming becomes inconsistent and developers waste time manually searching for the right icon. AntDesign offers an image‑based icon search, yet it only works for AntDesign icons, requires square screenshots, and cannot handle stretched images.

The solution is a custom, pure‑front‑end icon recognizer built on machine learning.

Key Terms

Machine Learning refers to algorithms that enable computers to learn from data and make predictions. It is an umbrella term for many algorithms such as linear regression, decision trees, and deep learning.

CNN (Convolutional Neural Network) is a deep, feed‑forward network specialized for visual image analysis. CNNs reduce high‑dimensional image data while preserving features, making them ideal for image classification tasks.

Sample Generation

Icons are rendered on a page, then captured using an off‑screen canvas. The canvas content is cropped to remove empty borders and resized to a square shape. By iterating over different icons and font sizes, a large JSON dataset of base64‑encoded images is produced.

// Draw icon on off‑screen canvas
offscreenCtx.font = `20px NextIcon`;
offscreenCtx.fillText(labelMap[labelName]);

// Compute crop coordinates
const { x, y, width: w, height: h } = getCutPosition(canvasSize, canvasSize, offscreenCtx.getImageData(0, 0, canvasSize, canvasSize).data);

function getCutPosition(width, height, imgData) {
  let lOffset = width, rOffset = 0, tOffset = height, bOffset = 0;
  // iterate pixels to find non‑empty bounds (omitted for brevity)
  return { x: lOffset, y: tOffset, width: rOffset - lOffset, height: bOffset - tOffset };
}

// Export as base64
ctx.drawImage(offscreenCanvas, x, y, w, h, 0, 0, 96, 96);
const dataUrl = canvas.toDataURL('image/jpeg');

The resulting JSON contains thousands of samples (e.g., 350 icons × ~70 sizes each).

Model Training

Pipcook, an open‑source ML toolkit for web developers, is used to train the model. The workflow includes data collection, preprocessing, model definition (MobileNet), training, and evaluation.

{
  "plugins": {
    "dataCollect": {
      "package": "@pipcook/plugins-image-classification-data-collect",
      "params": { "url": "file://path/to/icon_dataset.zip" }
    },
    "dataAccess": { "package": "@pipcook/plugins-pascalvoc-data-access" },
    "dataProcess": {
      "package": "@pipcook/plugins-tfjs-image-classification-process",
      "params": { "resize": [224, 224] }
    },
    "modelDefine": { "package": "@pipcook/plugins-tfjs-mobilenet-model-define" },
    "modelTrain": {
      "package": "@pipcook/plugins-image-classification-tfjs-model-train",
      "params": { "batchSize": 64, "epochs": 12 }
    },
    "modelEvaluate": { "package": "@pipcook/plugins-image-classification-tfjs-model-evaluate" }
  }
}

Run training with: $ pipcook run config.json After about two hours on a Mac, the model achieves ~98.5% accuracy with a loss of 0.05. The output folder contains model.json and weights.bin ready for front‑end use.

Model Usage

Load the TensorFlow.js model in the browser and preprocess incoming screenshots with the same cropping logic. Predict the top‑5 most similar icons.

import * as tf from '@tensorflow/tfjs';
const model = await tf.loadLayersModel('model.json');
// Crop image to square
const { x, y, width: w, height: h } = getCutPosition(imgW, imgH, offscreenCtx.getImageData(0, 0, imgW, imgH).data);
ctx.drawImage(offscreenCanvas, x, y, w, h, 0, 0, 224, 224);
const imgTensor = tf.browser.fromPixels(canvas).resizeBilinear([224, 224]).reshape([1, 224, 224, 3]);
const pred = model.predict(imgTensor).arraySync()[0];
const top5 = pred.map((score, i) => ({ score, label: labelArray[i] }))
               .sort((a, b) => b.score - a.score)
               .slice(0, 5);
console.log(top5);

The tool runs entirely in the browser—no backend service is required—making it suitable for static sites or component library documentation.

Conclusion

From code writing to a deployable model, the process takes a weekend plus two evenings, with most time spent on environment setup and training. Pipcook simplifies the workflow but documentation is sparse, so developers may need to read source code and experiment.

References

Stanford Machine Learning Course – Coursera

TensorFlow.js Official Site

Pipcook Documentation

Articles on CNN and Machine Learning