Hybrid AI Engine: Integrating On‑Device Image Recognition with TensorFlow Lite and HiAI

Artificial intelligence, deep learning, and machine learning are now widely used in mobile apps for tasks such as image beautification, automatic caption generation, voice‑to‑text conversion, and AI‑powered customer service.

Traditionally, there are three ways to bring machine‑learning results to the client side:

Solution 1: The client uploads compressed images to cloud storage, where a server‑side AI model performs object detection and returns a caption.

Solution 2: The client embeds a TensorFlow Lite (or similar) model and runs inference locally on the CPU.

Solution 3: The client calls a system‑level AI capability (e.g., Huawei HiAI) that uses dedicated NPU hardware.

Each solution has drawbacks: Solution 1 consumes network bandwidth and raises privacy concerns; Solution 2 increases power consumption, app size, and requires developers to be both AI and front‑end engineers; Solution 3 is limited to a few device models.

To address these issues, Sohu News introduced a Hybrid AI Engine that abstracts the underlying AI implementation. The framework exposes a unified AI interface, automatically selecting the best available backend (cloud, TensorFlow Lite, or system AI) and handling model download, updates, and result correction.

Using the engine, developers need only a few lines of code to obtain a caption from an image:

AIHelperFactory aiHelperFactory = AIHelperFactory.getInstance(context);
aiHelperFactory.init(AIHelperFactory.AI_TOOL_RECOGNIZER);
if (AIHelperFactory.getInstance(context).isReady(AIHelperFactory.AI_TOOL_RECOGNIZER)) {
    description = AIHelperFactory.getInstance(context)
        .getRecognizer().recognize(recognizeBitmap);
}
AIHelperFactory.getInstance(context).release(AIHelperFactory.AI_TOOL_RECOGNIZER);

In contrast, a comparable TensorFlow Lite implementation requires extensive boilerplate for model loading, image preprocessing, and result handling, as shown below:

private void initModel(Context context) {
    final Context appContext = context.getApplicationContext();
    ModelControler.getInstance(appContext).prepareModels(TF_MODEL_NAME, new ModelControler.ModelDownloadListener() {
        @Override public void onModelReady() {
            String modelPath = ModelControler.getInstance(appContext).getModelPath(TF_MODEL_NAME);
            recreateClassifier(appContext, modelPath);
        }
        @Override public void onModelDownloadFailed() { /* TODO: retry? */ }
    });
}

public void recreateClassifier(Context context, String modelPath) {
    if (mClassifier != null) { mClassifier.close(); mClassifier = null; }
    try {
        mClassifier = TFLiteObjectDetectionAPIModel.create(
            modelPath + TF_OD_API_MODEL_FILE,
            modelPath + TF_OD_API_LABELS_FILE,
            mInputSize,
            TF_OD_API_IS_QUANTIZED);
        mRecognizedBitmap = Bitmap.createBitmap(mInputSize, mInputSize, Bitmap.Config.ARGB_8888);
    } catch (IOException e) { e.printStackTrace(); }
}

private List<Recognition> recognizeQualify(Bitmap sourceBitmap) {
    if (mClassifier == null || mRecognizedBitmap == null) return null;
    Matrix frameToCropTransform = ImageUtils.getTransformationMatrix(
        sourceBitmap.getWidth(), sourceBitmap.getHeight(), mInputSize, mInputSize, 0, true);
    Matrix cropToFrameTransform = new Matrix();
    frameToCropTransform.invert(cropToFrameTransform);
    Canvas canvas = new Canvas(mRecognizedBitmap);
    canvas.drawBitmap(sourceBitmap, frameToCropTransform, null);
    List<Recognition> results = mClassifier.recognizeImage(mRecognizedBitmap);
    List<Recognition> qualifyResults = new ArrayList<>();
    for (Recognition r : results) {
        if (r.getConfidence() >= MINIMUM_CONFIDENCE_TF_OD_API) qualifyResults.add(r);
    }
    return qualifyResults.isEmpty() ? null : qualifyResults;
}

The Hybrid engine first checks for a system‑level AI implementation; if unavailable, it falls back to TensorFlow Lite, downloading and updating models as needed. Initialization looks like this:

public static AIHelperFactory getInstance(Context context) {
    if (sInstance != null) return sInstance;
    sInstance = new AIHelperFactory(context);
    return sInstance;
}

public void init(final int... tools) {
    if (HiAIUtils.isHuaweiAIAppInstalled(mContext)) {
        VisionBase.init(mContext, new ConnectionCallback() {
            @Override public void onServiceConnect() { for (int tool : tools) createTools(tool); }
            @Override public void onServiceDisconnect() { release(); }
        });
    } else {
        for (int tool : tools) createTools(tool);
    }
}

Beyond image captioning, the engine already supports face detection, document recognition, and speech input, with each module packaged as an optional 100 KB SDK that can be downloaded on demand, keeping the base APK lightweight.

Future plans include open‑sourcing the entire Hybrid AI Engine, adding more plug‑in modules, and further simplifying on‑device AI development for mobile engineers.