The Story Behind the Creation of UTF-8 and Its Advantages

In September 1992, Rob Pike was finalizing Plan 9 at Bell Labs when IBM called to ask for a review of a new Unicode encoding. Together with Ken Thompson, they saw an opportunity to design a better Unicode storage standard.

They proposed a fast, high‑quality solution that could be completed within three days. By the following Friday, Plan 9 was running on UTF‑8, and the implementation quickly became the de‑facto standard for the Web (now used by over 96% of sites).

Unicode defines code points (e.g., the character “码” is U+7801, binary 111 1000 0000 0001) but does not prescribe how to store them. Early encodings used fixed two‑byte representations, wasting space for ASCII characters.

UTF‑8 solves this by using a variable‑length scheme: one byte for ASCII, up to four bytes for other characters. The first byte indicates the total length, allowing parsers to determine character boundaries instantly.

Key advantages of UTF‑8 include:

1. Compatibility with ASCII – the highest bit of multibyte characters is always 1, while ASCII’s highest bit is 0, preventing conflicts.

2. Length prefix – the leading bits of the first byte tell how many continuation bytes follow, simplifying decoding.

3. Prefix‑free property – no valid character is a prefix of another, enabling error‑resilient processing and easy skipping of corrupted bytes.

These design choices made UTF‑8 both efficient and robust, leading to its widespread adoption across the Internet.