How Chinese Characters Are Stored in Computers: Unicode & UTF-8 Deep Dive

The Era of Encoding Chaos

Before Unicode, each country and region developed its own encoding schemes:

China : GB2312, GBK, GB18030

Japan : Shift-JIS, EUC-JP

Korea : EUC-KR

Taiwan : Big5

These encodings had different character sets and rules. The problem: opening a file with the wrong encoding produces garbled text (mojibake). For example, a file written in GBK but opened with Shift-JIS shows completely different characters because the same byte sequences map to different characters in each encoding.

Unicode: One Encoding to Rule Them All

Unicode assigns a unique number, called a code point , to every character in every writing system — English, Chinese, Japanese, Korean, Arabic, emoji, even extinct scripts. Examples:

A → U+0041

中 → U+4E2D

国 → U+56FD

🎉 → U+1F389

🐶 → U+1F436

The "U+" prefix indicates a code point in hexadecimal. With Unicode, every character has a universal "ID card", eliminating cross-encoding conflicts.

How Unicode Is Stored

Unicode defines hundreds of thousands of code points. How are they stored in computers?

UTF-32: The Straightforward Approach

UTF-32 uses 4 bytes (32 bits) per character. Simple but wasteful: the ASCII letter "A" needs only 1 byte, yet UTF-32 uses 4 bytes, mostly zeros.

UTF-16: A Compromise

UTF-16 uses 2 or 4 bytes. Most common characters fit in 2 bytes; rare characters use 4. Windows and Java use UTF-16 internally.

UTF-8: The Most Popular Scheme

UTF-8 is a variable-length encoding :

English letters : 1 byte (e.g., A, b, 0, !)

Latin extended : 2 bytes (e.g., é, ñ, ü)

CJK characters : 3 bytes (e.g., 中, 日, 한)

Emoji and supplementary planes : 4 bytes (e.g., 🎉, 🐶, 👍)

UTF-8's clever design:

Full ASCII compatibility : All ASCII characters have identical UTF-8 and ASCII encodings.

No mojibake : Each character's encoding is unique and never overlaps with another.

Space efficiency : English uses 1 byte, Chinese uses 3 bytes — far more efficient than fixed-length schemes.

Because of these advantages, UTF-8 became the dominant encoding on the web; most web pages and apps use UTF-8.

UTF-8 Encoding Rules

UTF-8 uses specific bit patterns to indicate byte length:

Single-byte (ASCII): starts with 0 → 0xxxxxxx Two-byte: first byte starts with 110, continuation bytes start with 10 → 110xxxxx 10xxxxxx Three-byte: first byte starts with 1110 → 1110xxxx 10xxxxxx 10xxxxxx Four-byte: first byte starts with 11110 → 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx Pattern summary:

Byte starting with 0 → single-byte character

Byte starting with 10 → continuation byte of a multi-byte character

Byte starting with 11 → leading byte of a multi-byte character

This design lets UTF-8 self-synchronize: even if data is corrupted, the next character boundary can be found quickly.

What Chinese Looks Like in UTF-8

Take the character "中" (U+4E2D). Its binary is 0100 1110 0010 1101 (16 bits). Since it needs 16 bits, it uses the 3-byte UTF-8 template: 1110xxxx 10xxxxxx 10xxxxxx Fill the 16 bits into the x positions: 1110|0100 10|111000 10|101101 Result: E4 B8 AD (hexadecimal). So "中" occupies 3 bytes: E4, B8, AD.

Why Mojibake Happens

Mojibake occurs when data is decoded with the wrong encoding. Common scenarios:

File is UTF-8, opened as GBK → Chinese becomes garbled

Web page is GBK, browser uses UTF-8 → Chinese becomes garbled

Email encoding mismatch → entire message garbled

Classic examples: ä½ å¥½ → UTF-8 "你好" interpreted as Latin-1 浣犲ソ → UTF-8 "你好" interpreted as GBK ??? → completely unrecognized characters

Solution: ensure encoding and decoding use the same scheme. Today UTF-8 is standard almost everywhere, so mismatches are rare.

BOM: The File's Self-Introduction

Some text files start with a BOM (Byte Order Mark). UTF-8's BOM is EF BB BF (three bytes), signaling "this file is UTF-8". UTF-8 doesn't require BOM for detection, so many programs omit it. UTF-16 and UTF-32 usually need BOM because of byte-order issues.

Encoding Best Practices in Programming

Save source code files as UTF-8 .

Explicitly specify encoding when reading/writing files.

Use UTF-8 for network transmission.

In MySQL, use utf8mb4 (the old utf8 only supports 3-byte characters; utf8mb4 supports full 4-byte emoji).

Key Takeaways

Unicode is a character set : assigns a unique number to every character.

UTF-8 is an encoding scheme : converts those numbers into byte sequences for storage and transmission.

Next time you see mojibake, you'll know exactly where the mismatch occurred.