Mastering Attention: Build a Transformer-Based Text Recognition Network

Topic: Attention-based Text Recognition Network

Speaker: Yao Zhuokun (Image Algorithm Junior Engineer)

1. From Attention to Transformer

Attention mechanisms play a pivotal role in NLP and computer vision. Before attention, Seq2Seq models relied on an encoder‑decoder architecture with a fixed‑size context vector, which limited expressive power.

Attention generates dynamic input vectors for each output time step, overcoming the bottleneck of a single context vector.

The time‑varying input \(c_t\) can be expressed as:

The general Attention formula is:

Here \(Q, K, V\) denote query, key, and value. Self‑Attention, a special case where \(Q, K, V\) come from the same sequence, is illustrated below:

Transformer units incorporate positional encoding, residual connections, layer normalization, and feed‑forward layers, with Multi‑head Attention as the core component.

2. Transformer Code Implementation

The Scaled Dot‑Product Attention is implemented as a matrix multiplication, enabling parallel computation:

Building on this, Multi‑head Attention splits \(Q, K, V\) into several heads, applies Scaled Dot‑Product Attention to each, and then concatenates the results:

Multi‑head Self‑Attention and the full Transformer unit are realized as:

3. Transformer‑Based Text Recognition Network

Integrating the Transformer unit into a text recognition pipeline yields a three‑stage architecture that progressively refines visual and semantic features.

The first stage extracts CNN features and enhances them with a Transformer to produce the initial output. The second stage embeds the argmax text sequence and applies another Transformer to capture richer semantic information. The third stage adds a diagonal mask in the Transformer to enforce a “fill‑in‑the‑blank” behavior, further emphasizing semantic cues.

The final output fuses visual, weak semantic, and strong semantic features, achieving efficient multimodal utilization. The complete implementation code is illustrated below: