Build an AI‑Powered Draw‑and‑Guess Game with TensorFlow.js on the Frontend

Introduction

In the AI era, frontend developers can leverage on‑device AI models to create interactive experiences such as an AI‑powered “draw‑and‑guess” game.

AI Quickdraw Gameplay Overview

The game presents a prompt word; the player draws it on a canvas, and an AI model predicts the drawing. If the prediction matches the prompt, the player scores points.

Training the Model

We use the Google QuickDraw dataset (optionally a subset of 80 classes) and TensorFlow.js to build a CNN classifier. The tutorial explains data loading, model definition, training loop, and saving the model files ( model.json, weights.bin, classes.json).

import * as tf from "@tensorflow/tfjs-node";
import { DoodleData } from "./model/doodle-data.model";

export class Classifier {
  constructor(data: DoodleData) {
    this.data = data;
    this.model = tf.sequential();
    this.model.add(tf.layers.conv2d({
      inputShape: [data.IMAGE_WIDTH, data.IMAGE_HEIGHT, 1],
      kernelSize: 3,
      filters: 16,
      strides: 1,
      activation: "relu",
      kernelInitializer: "varianceScaling"
    }));
    this.model.add(tf.layers.maxPooling2d({ poolSize: [2, 2], strides: [2, 2] }));
    this.model.add(tf.layers.conv2d({ kernelSize: 3, filters: 32, strides: 1, activation: "relu", kernelInitializer: "varianceScaling" }));
    this.model.add(tf.layers.maxPooling2d({ poolSize: [2, 2], strides: [2, 2] }));
    this.model.add(tf.layers.flatten());
    this.model.add(tf.layers.dense({ units: data.totalClasses, kernelInitializer: "varianceScaling", activation: "softmax" }));
    const optimizer = tf.train.adam();
    this.model.compile({ optimizer, loss: "categoricalCrossentropy", metrics: ["accuracy"] });
  }

  async train() {
    const trainingData = tf.data.generator(() => this.data.dataGenerator("train")).shuffle(this.data.maxImageClass * this.data.totalClasses).batch(64);
    const testData = tf.data.generator(() => this.data.dataGenerator("test")).shuffle(this.data.maxImageClass * this.data.totalClasses).batch(64);
    await this.model.fitDataset(trainingData, { epochs: 5, validationData: testData });
  }

  async save() {
    fs.mkdirSync("doodle-model", { recursive: true });
    fs.writeFileSync("doodle-model/classes.json", JSON.stringify({ classes: this.data.classes }));
    await this.model.save("file://./doodle-model");
  }
}

Integrating the Model into the Page

After training, the model files are placed in the public assets folder. DoodleClassifier.js loads the model and provides loadModel, predictTopN, and predict methods.

import * as tf from "@tensorflow/tfjs";
import apiClient from "@/services/http";

export default class DoodleClassifier {
  async loadModel() {
    this.model = await tf.loadLayersModel("assets/doodle-model/model.json");
    const response = await apiClient.get("assets/doodle-model/classes.json");
    this.classes = response.data.classes;
  }

  async predictTopN(data, n) {
    const predictions = Array.from(await this.model.predict(data).data());
    const indexed = predictions.map((p, i) => ({ probability: p, index: i }));
    indexed.sort((a, b) => b.probability - a.probability);
    return indexed.slice(0, n).map(p => ({ label: this.classes[p.index], accuracy: p.probability }));
  }

  async predict(data) {
    const argMax = await this.model.predict(data).argMax(-1).data();
    return this.classes[argMax[0]];
  }
}

The Vue component DoodleView.vue creates a canvas, captures its pixel data, normalizes it, and calls the classifier to obtain the top‑N predictions.

const tensor = tf.browser.fromPixels(imgData, 1);
const resized = tf.image.resizeBilinear(tensor, [28, 28])
  .reshape([1, 28, 28, 1])
  .toFloat();
const normalized = tf.scalar(1.0).sub(resized.div(tf.scalar(255.0)));
this.model.predictTopN(normalized, 5).then(predictions => { this.predictions = predictions; });

Optimization Measures

Standardize canvas input by center‑cropping and thickening strokes, and move inference to a Web Worker to keep the UI responsive.

Conclusion

The end‑to‑end example shows how frontend engineers can adopt on‑device AI for low‑latency, privacy‑preserving interactions, while also noting that cloud‑based APIs may be preferable when resources allow.

References

RiccardoGai/doddle-classifier-model

RiccardoGai/doddle-classifier-app

yining1023/doodleNet