Boosting ID Card Photo Quality with Multimodal AI: A Practical Deployment Guide

Introduction

This article introduces a practice of applying a multimodal AI model for ID card photo quality detection. When users upload ID photos, common quality problems cause OCR failures; the multimodal model provides intelligent detection and friendly feedback to guide users to re‑upload qualified photos.

Business Background

With the rapid development of internet services, identity verification has become essential. In the Taobao Finance ("Scene Finance") project, analysis of uploaded ID photos revealed several frequent quality issues that interrupt user flows.

Common Photo Quality Issues

Non‑ID image – uploading unrelated documents such as social security cards.

Side order error – selecting the wrong side of the ID.

Blur – unreadable key information.

Reflection – over‑exposed areas.

Occlusion – key fields blocked by fingers or objects.

Incomplete – cropped edges or wrong proportions.

Multiple cards – uploading both sides as one image.

These problems prevent normal information extraction, causing users to abandon the process; the goal is to encourage higher‑quality uploads.

Why Use a Multimodal Model for Image Detection?

The current OCR service is provided by Alibaba Cloud OCR. The multimodal model from Alibaba Cloud Bailei platform is introduced to perform photo‑quality detection when OCR fails, delivering user‑friendly prompts.

Differences Between OCR and Multimodal Text Extraction

Deep‑learning OCR – optimized for character recognition, focusing on detecting and recognizing text in images. It is used for specific scenarios such as ID cards, invoices, and license plates.

Multimodal model text extraction – large‑scale pretrained models (e.g., GPT‑style) that understand both image and text, handling complex contexts, supporting multiple document types, and generating structured outputs like JSON or XML.

Key Advantages of Multimodal Models

Recognize various document types and their elements (driver’s license, student ID, invoices, etc.).

Return results in custom formats (JSON, XML, tables) and support language switching.

Flexible quality‑checking rules adaptable to specific business scenarios.

Generate user‑facing suggestions based on detection results.

Model Call Issues and Solutions

Hallucination – the model may fill missing information incorrectly (e.g., wrong dates or locations).

High latency – average response time around 3 seconds, which can affect user experience.

Accuracy variance – different models (genmini, GPT, Qwen) show large accuracy gaps.

Stability – need fallback handling for service instability or unexpected responses.

Solutions include: keep stable OCR for data extraction, use asynchronous model calls when OCR succeeds, synchronous calls with strong prompts when OCR fails, continuously optimize prompts, and monitor with feature flags and alerts.

Release Strategy and Online Performance

The AI feature was rolled out using a three‑stage strategy: “no‑perception pre‑release”, “progressive”, and “gray‑scale” rollout, gradually increasing traffic from 1 % to 100 %.

Online metrics show average response time stable at ~3 seconds, no service exceptions, and the distribution of quality issues: “non‑ID” and “side order error” together exceed 90 %, “blur/reflection” 5.4 %, “incomplete/occlusion” 2.7 %, “multiple cards” 0 %.

User‑Friendly Prompt Texts

Non‑ID image: "The uploaded image does not meet ID‑photo requirements. Please upload a valid ID photo."

Wrong side: "The uploaded photo does not match the selected type. Please upload the correct side."

Occlusion: "Key information is blocked. Please ensure it is visible and re‑upload."

Blur: "The image is blurry. Please take a clear photo and re‑upload."

Multiple cards: "Multiple IDs detected. Please upload a single, complete ID photo."

Fallback: "ID recognition failed. Please re‑upload."

Cost Estimation of Bailei Model Calls

Cost is calculated by input and output token counts multiplied by the model’s per‑token price. The estimated unit price is around 0.01 CNY per call.

Business and Technical Value

AI application breakthrough – first AI integration in the scenario‑finance product.

Improved conversion and entry rates by guiding users to upload better photos.

Scenario adaptability – prompt‑driven large‑model usage enables quick expansion to other document types.

Reusability – unified Mtop interface reduces code duplication across products.

Team Introduction

Author: Wei Xi, Scene Finance Technology team, Taobao Group. The team focuses on building trustworthy financial services for SMEs within the Alibaba ecosystem, providing end‑to‑end financial solutions to improve transaction efficiency and trust.