How a Big Data Platform Powers Real‑Time Facial Recognition for Billion‑Scale Face Libraries

Case Overview

Case name: 恒远华动态人脸采集与实时识别方案 (Dynamic Face Capture and Real‑Time Recognition Solution)

Source: Beijing 恒远华信息技术有限公司

Industry: Other

Technology domain: Big Data

Huawei product: FusionInsight HD

Key Advantages of Using FusionInsight

Supports massive data processing, capable of handling a face library of hundreds of billions of entries.

Provides integrated storage, processing, and analysis.

Real‑time capability: storm‑based deep‑learning algorithms enable instant detection, comparison, and search.

High efficiency: duplicate‑checking time reduced to one‑tenth of the original.

Strong security for network, data, and host layers.

Business Background

Facial recognition is a biometric technology that identifies individuals by analyzing facial features captured via cameras or video streams. It includes detection, tracking, feature extraction, and matching against stored face libraries.

Challenges with Traditional Systems

Users can only obtain video data from third‑party sources, not raw face photos.

Need to extract faces from video and perform real‑time matching simultaneously.

Camera angles of existing devices often fail to capture usable face images.

Requirement to build multi‑business face libraries reaching billions of records.

Large‑scale duplicate checks previously took months per operation.

Solution Overview

By applying deep‑learning facial‑recognition algorithms on Huawei’s big‑data platform, the solution resolves the above issues and dramatically improves system performance.

System Core Functions

Real‑time reception: Capture face photos from external devices and store them in HDFS.

Real‑time face detection: Detect faces, extract features, and store them in a capture library linked with identity data.

Real‑time blacklist comparison: Compare extracted features with a blacklist and trigger alerts on matches.

Offline face recognition: Match submitted photos against registration, blacklist, or capture libraries.

Face‑library management: Create, update, and delete entries in capture, registration, and blacklist libraries.

User interface: Web portal for library management, offline search, capture record query, trajectory view, and blacklist alerts.

System Design Performance

Registration library supports 10 million records with smooth scaling to 100 billion.

Offline retrieval time under 10 seconds.

From photo receipt to blacklist comparison under 2 seconds.

Capture library content can be migrated to the registration library as needed.

System Logical Architecture

The architecture consists of three main components:

Face capture system: Professional capture cameras or HD network cameras plus a detection server to extract faces from video streams and associate them with identity information.

Face matching system: Performs feature extraction, builds face models, and conducts real‑time matching and post‑event retrieval.

Face libraries: Capture library (raw images and features), registration library (standard faces with identity), and blacklist library (high‑risk individuals for real‑time alerts).

Data Flow and Processing

Two primary data flows exist: real‑time video face matching and offline image search. Real‑time matching alerts operators when a monitored individual appears; offline queries enable post‑event investigations.

Huawei Big Data Platform Components

FusionInsight provides a unified suite for large‑scale data storage, query, and analysis, comprising:

FusionInsight HD – distributed data processing and real‑time streaming.

FusionInsight MPPDB – column‑store MPP relational database for PB‑scale structured data.

FusionInsight Miner – data mining and analytics.

FusionInsight Farmer – container platform for big‑data applications.

FusionInsight Manager – cluster operation and maintenance.

Underlying technologies include HDFS for distributed file storage, MapReduce for batch processing, YARN for resource management, HBase for column‑oriented NoSQL storage, Apache Storm for real‑time stream processing, and Kafka for high‑throughput messaging.

Implemented APIs

Java APIs used in the solution:

HBaseAdmin, HTableDescriptor, HTable, Put, Get – for table creation, deletion, and CRUD operations.

HDFS Java API – to read photo files stored in HDFS.

Kafka producer and consumer APIs – to ingest face photos into Storm spouts for real‑time capture.

Storm API – to submit, query, and delete topologies.

Partner Introduction

Beijing 恒远华信息技术有限公司, founded in 2001, specializes in networking and information‑technology solutions for government and large enterprises. It has long‑term partnership with Huawei and has delivered multiple national‑level communication projects. With the growth of Huawei’s big‑data platform, the company developed this facial‑recognition solution for industry use.