Applying AI Technologies in the Youdao Dictionary Pen: Scanning, Offline Translation, and Edge ML Library

The Youdao Dictionary Pen is a compact smart hardware device designed to help students look up words, translate text, and learn English, featuring a wide‑angle camera for scanning and a high‑definition display for interactive learning.

01 Youdao Dictionary Pen Introduction

The pen’s tip houses a large‑angle camera that captures images of text for input, while the body displays queried words or translations and supports interactive picture reading.

02 Scanning and Point‑Query

Scanning differs from typical OCR because the camera captures a continuous stream of small windows at about 100 frames per second. A panoramic stitching algorithm extracts the relevant text lines from these frames in three stages: pixel‑level detection, central line grouping, and line correction, achieving roughly 98% accuracy.

The system also corrects wide‑angle distortion and uneven lighting by pre‑capturing calibration images, applying inverse geometric transforms, and enhancing illumination before OCR.

03 Offline Translation

To support offline use cases such as classrooms or travel, the translation service is deployed on‑device, eliminating network latency and preserving user privacy.

Because the original online NMT model contains over a hundred million parameters, several compression techniques are applied:

Depth and width pruning (favoring encoder layers, resulting in a 5‑layer encoder and 3‑layer decoder).

Embedding sharing across source and target vocabularies, reducing the embedding parameter count by about half.

Shared attention across layers to reuse query/key weights.

Quantization from float32 to int8 using linear mapping, dramatically reducing memory and compute.

Knowledge distillation, transferring the performance of a large teacher model to a smaller student model.

These methods together maintain translation quality (BLEU drop ≤ 0.1) while cutting memory usage by 50‑60% and increasing inference speed by 45‑67%.

04 EMLL (Edge ML Library)

EMLL is a high‑performance edge‑side machine‑learning library designed to accelerate flat‑matrix computations common in on‑device AI. It supports fp32, fp16, and int8 data types and provides assembly‑level optimizations for ARM Cortex‑A35/A53/A55/A76/A77 cores.

Benchmarks show that EMLL delivers 30%‑100% speedup over Eigen and ARM Compute Library on typical OCR, ASR, and NMT workloads, reducing latency and enabling larger models on low‑power devices.

The library is open‑source (https://github.com/netease-youdao/EMLL) and works on Linux and Android.

05 Q&A

Q: Does sharing target‑side embedding matrices increase decoding time? A: In practice the shared vocabulary is trimmed, and with EMLL the impact on speed is negligible.

Q: Can parameter sharing, quantization, and distillation be used together? A: Yes. Sharing has little speed impact, quantization speeds up inference, and distillation improves quality without affecting speed.

Finally, the Youdao team is recruiting algorithm, product, and functional engineers; interested candidates can contact Zhang Guang Yong ([email protected]).