Skip to content
How Ken Thompson Designed UTF-8 on a Placemat to Save C and Unix from 16-bit Unicode
Fundamentals 24 min read

How Ken Thompson Designed UTF-8 on a Placemat to Save C and Unix from 16-bit Unicode

In 1992, Ken Thompson and Rob Pike invented UTF-8 on a restaurant placemat to replace the flawed 16-bit UCS-2 standard, creating a self-synchronizing, ASCII-compatible encoding that preserved C strings and Unix pipelines, ultimately becoming the dominant character encoding for the web and modern languages.

Radish, Keep Going!
Radish, Keep Going!
Radish, Keep Going!

In the 1980s, engineers handling non-English text endured a two-decade nightmare. The industry foundation was the 1963 ASCII encoding — only 7 bits, 128 characters. As PCs spread to Western Europe, Japan, and East Asia, vendors patched the high bit or used double-byte schemes, creating hundreds of incompatible code pages: DOS CP437, Western Europe ISO-8859-1, Traditional Chinese Big5, Simplified Chinese GB 2312, Japanese Shift-JIS.

Text files carried no metadata about their code page. A document written in Tokyo opened in Germany turned into mojibake; high bytes sometimes collided with terminal escape sequences, making cursors jump and speakers beep.

Teletype Model 33 ASR teleprinter exhibited at Computer History Museum
Teletype Model 33 ASR teleprinter exhibited at Computer History Museum

Teletype Model 33 ASR teleprinter. ASCII was originally designed for this machine in 1963; its 7-bit, 128-character set underpinned three decades of Unix and C. (Photo by ArnoldReinhold, CC BY-SA 3.0, via Wikimedia Commons)

To end the chaos, engineers from Xerox, Apple, Microsoft, and others formed the Unicode Consortium in 1988, aiming to assign a unique number to every human character. To avoid variable-length parsing overhead, the committee proposed a fixed 16-bit encoding (UCS-2), believing 65,536 code points would suffice. Fixed width would allow O(1) random access via buffer[i * 2] and simplify layout engines.

By 1992, as UCS-2 neared ratification as ISO 10646, it planted a time bomb under Unix and C.

The Committee's Blind Spot: How a 0x00 Byte Could Blow Up Unix Pipes

UCS-2 forced every ASCII character into two bytes, inserting a null byte 0x00 next to each letter. The string "cat" on a SPARC (big-endian) machine became 0x00 0x63 0x00 0x61 0x00 0x74 0x00 0x00; on x86 (little-endian) it became 0x63 0x00 0x61 0x00 0x74 0x00 0x00 0x00.

Standard 128-character ASCII table
Standard 128-character ASCII table

Standard 128-character ASCII table. All traditional C string functions assume 0x00 as terminator; the high-byte 0x00 introduced by fixed 16-bit encoding shattered that foundation. (Public Domain, via Wikimedia Commons)

That embedded 0x00 broke Dennis Ritchie's 1972 C contract: C strings have no length prefix; every standard library function relies on an in-memory \0 terminator. On little-endian, strlen() stops at the 0x00 after 'c', returning length 1; on big-endian, the very first byte is 0x00, returning length 0.

Worse, the Unix kernel treats paths as opaque byte streams, recognizing only ASCII 0x2F ('/') as directory separator. If a multi-byte character contained 0x2F or 0x00, non-English filenames would be truncated or misparsed as deep subdirectories.

The committee's patch was wide-character APIs ( wchar_t, wcslen, wcscpy). But fragmentation followed: Microsoft made wchar_t 16-bit in Windows NT; most Unix vendors made it 32-bit. Billions of lines of existing C code, network parsers, and shell pipelines could not be rewritten overnight.

The September 2, 1992 Phone Call and the Division-Based Proposal

That summer, Ken Thompson and Rob Pike at Bell Labs were building the distributed OS Plan 9, which needed native multilingual support. The early ISO proposal was so painful they abandoned it.

AT&T Bell Labs Murray Hill, New Jersey building
AT&T Bell Labs Murray Hill, New Jersey building

Bell Labs Murray Hill campus. On the afternoon of September 2, 1992, X/Open representatives called the Computer Science Research Center here seeking Thompson and Pike's review of a multi-byte encoding proposal. (Photo by United States Department of Energy, Public Domain, via Wikimedia Commons)

On September 2, Rob Pike took a call from Austin, Texas — X/Open representatives discussing the File System Safe UTF (FSS-UTF) proposal drafted by Dave Prosser. Reviewing the draft, Thompson and Pike found serious engineering flaws. Based on archives later uncovered by Russ Cox, Prosser's implementation used hard-coded offset constants and relied heavily on constant subtraction and integer division:

#define OFF1  0x0000080
#define OFF2  0x0002080

int wctomb(char *s, wchar_t wc)
{
    if (wc <= 0x7f) {
        s[0] = wc;
        return 1;
    }
    if (wc <= 0x1fff + OFF1) {
        wc -= OFF1;
        s[0] = 0x80 | (wc >> 7);
        s[1] = 0x80 | (wc & 0x7f);
        return 2;
    }
    /* ... more constant subtraction and multi-level branches ... */
}

Beyond extra arithmetic, the core problem was lack of self-synchronization in the data stream . In Prosser's scheme, both lead and continuation bytes used the high-bit pattern 1xxxxxxx, so their ranges overlapped. If a byte was lost on the network or a program lseek() ed into the middle of a file, the decoder could not tell whether it was at a character start or tail, causing thousands of subsequent characters to misalign.

Pike flagged the defect. X/Open replied: final vote is Monday, September 7. If Bell Labs could deliver a working alternative before then, they would adopt it.

The two men hung up and went to dinner.

Bit Masks on a Placemat: Why the Math Guarantees Self-Synchronization

That evening, in a New Jersey diner, Ken Thompson grabbed a paper placemat and derived the bit layout using only Boolean logic and prefix masks — no arithmetic constants, no division. The rules that became UTF-8:

0xxxxxxx                         ASCII (0x00-0x7F), 1 byte
110xxxxx 10xxxxxx                2-byte sequence
1110xxxx 10xxxxxx 10xxxxxx       3-byte sequence
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 4-byte sequence

This structure simultaneously solves compatibility and fault tolerance through strict mathematical invariants:

Zero-break ASCII compatibility. When the lead byte's high bit is 0, it is exactly 7-bit ASCII ( 0x00–0x7F). All existing English source code and config files are valid UTF-8 unchanged. In multi-byte sequences, every byte has its high bit set to 1 ( 0x80–0xFF), so 0x00 and 0x2F ('/') never appear inside a multi-byte character, protecting C string functions and Unix path parsing.

Orthogonal, deterministic self-healing for continuation bytes. All continuation bytes share the prefix 10 ( 0x80–0xBF), a range disjoint from ASCII ( 0x00–0x7F) and all lead bytes ( 0xC0–0xFF). A decoder can jump into the stream at any point: if it sees 10xxxxxx it knows it is inside a character; skipping forward at most three bytes guarantees a non- 10 byte, re-establishing character boundaries.

Monotonic sort order and endianness independence. High-order bits come first, so C's native strcmp() on raw bytes yields the same ordering as comparing 32-bit Unicode code points. UTF-8 is a pure byte stream — no big-endian/little-endian ambiguity, no need for a BOM ( 0xFEFF) header.

The derivation finished just as the waiter brought their food.

Typical 1970s New Jersey diner
Typical 1970s New Jersey diner

Skyline Diner, New Jersey, 1978. On the evening of September 2, 1992, Thompson and Pike derived UTF-8's bit masks on a paper placemat in a diner like this. (Photo by Acabashi, CC BY-SA 4.0, via Wikimedia Commons)

Ken Thompson and Dennis Ritchie at a PDP-11 in 1973
Ken Thompson and Dennis Ritchie at a PDP-11 in 1973

Ken Thompson (seated) and Dennis Ritchie at a PDP-11, 1973. Nineteen years later, 49-year-old Thompson again defined the multilingual character-stream rules for global C and Unix ecosystems in the span of a dinner. (Public Domain, via Wikimedia Commons)

48-Hour Rewrite of Plan 9: A System That Ran on Compatibility Alone

Back at Murray Hill near midnight, system logs show Thompson committed the first prototype at 23:44:10 EDT. He wrote the encode/decode bitwise core; Pike overhauled Plan 9's graphics window system and core libraries.

At 03:37:39 EDT on Friday, September 4, Thompson emailed Pike:

From ken Fri Sep 4 03:37:39 EDT 1992
you might want to look at
    /usr/ken/utf/utf.c
and see if you can make it prettier.

At 19:51:55 EDT that day, the pure-bitwise rewrite of /sys/src/libc/port/rune.c was merged. In this commit, Thompson formally named the system-layer Unicode abstraction rune — a term that later became a Go language cornerstone.

Over the weekend, batch scripts converted every text file, manual, compiler source, and font library in Plan 9 to the new encoding. Because the new encoding was 100% compatible with ASCII and \0, the transition was flawless: the assembler emitted no errors, kernel path parsing was unaffected, the sam editor smoothly opened code mixed with Greek letters.

By Monday, September 7, Plan 9 became the world's first OS running natively on UTF-8.

Plan 9 Acme windowed editor interface
Plan 9 Acme windowed editor interface

Plan 9's Acme programming environment and terminal. On September 4, 1992, Thompson committed rune.c; within two days the entire Plan 9 GUI, fonts, and system libraries switched natively to UTF-8. (Photo by Bell Labs, CC BY-SA 4.0, via Wikimedia Commons)

At 03:22:07 EDT on Tuesday, September 8, Thompson emailed the X/Open committee (cc'ing Dennis Ritchie):

From ken Tue Sep 8 03:22:07 EDT 1992
To: [email protected]
Cc: rob, dmr

Here is our modified FSS-UTF proposal. The words are the same as on
the previous proposal. My apologies to the author. The code has been
tested to some degree and should be pretty good shape. We have
converted Plan 9 to use this encoding and are about to issue a
distribution to an initial set of university users.

Faced with a production-grade, fault-free implementation and self-synchronization test data running on a full distributed OS, X/Open discarded the division-based draft and adopted the revision wholesale. In January 1993, Rob Pike and Peter Weinberger presented the paper at USENIX Winter; IETF subsequently standardized it as UTF-8 .

Rob Pike historical photo
Rob Pike historical photo

Unix pioneer, UTF-8 co-inventor, and Go language creator Rob Pike. On September 2, 1992, he insisted to X/Open that a data stream must be able to self-synchronize mid-stream — a rigorous engineering principle. (Photo by Original uploader, CC BY-SA 3.0, via Wikimedia Commons)

The Cost of UCS-2: Windows, Java, and JavaScript's Thirty-Year Technical Debt

While Bell Labs validated UTF-8, commercial giants bet heavily on the 16-bit fixed-width world.

Windows NT 3.1 (1993) bound system calls and NTFS to 16-bit UCS-2.

Java (1995) made its native char type 16-bit.

JavaScript (1995) built strings directly on 16-bit code units.

All assumed the character set would never exceed 65,536. That assumption lasted three years. As CJK historical characters were catalogued and mobile internet exploded with tens of thousands of emoji, Unicode code points blew past the 16-bit ceiling. Unicode 2.0 (1996) carved out 2,048 surrogate code points ( 0xD800–0xDFFF ) to pair two 16-bit units for supplementary characters, creating UTF-16 .

The fixed-width advantage vanished instantly: a character could be 2 or 4 bytes, str[i] random access degraded to sequential scan; 0x00 null bytes and BOM headers remained baggage. Today's JavaScript oddity "𠮷".length === 2 is a direct legacy of that compromise.

Unicode Consortium official logo
Unicode Consortium official logo

Unicode Consortium logo. After fixed 16-bit failure and surrogate-pair compromise, Unicode and W3C now enshrine UTF-8 as the sole universal character interchange foundation for the internet. (Public Domain, via Wikimedia Commons)

The web's explosion cemented the outcome. HTML, CSS, and code are ASCII-based; UTF-16 doubles transmission size, while UTF-8 adds zero overhead. Today, over 98.2% of the top 10 million websites use UTF-8.

In 2009, Thompson and Pike at Google designed Go with UTF-8 baked into Chapter 1, Section 1: Go source files must be UTF-8; the built-in string is a read-only byte slice, and the 32-bit rune was formally introduced. Modern Rust likewise mandates that String and &str be valid UTF-8.

The committee tried to impose fixed-width wide characters by top-down administrative force; a systems engineer's bitwise rules, scribbled on a placemat, won by doing zero harm to the existing ecosystem — becoming the unshakeable bedrock of human digital civilization.

References

[1] https://commons.wikimedia.org/wiki/File:ASR-33_at_CHM.agr.jpg

[2] https://commons.wikimedia.org/wiki/File:ASCII-Table-wide.svg

[3] https://commons.wikimedia.org/wiki/File:AT%26T_Murray_Hill.jpg

[4] https://commons.wikimedia.org/wiki/File:Skyline_Diner_New_Jersey_1978_soft_contrast.jpg

[5] https://commons.wikimedia.org/wiki/File:Ken_Thompson_and_Dennis_Ritchie--1973.jpg

[6] https://commons.wikimedia.org/wiki/File:Plan_9_from_Bell_Labs_(with_acme).png

[7] https://commons.wikimedia.org/wiki/File:Rob-pike.jpg

[8] https://commons.wikimedia.org/wiki/File:New_Unicode_logo.svg

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

C languageUnixUnicodeUTF-8character encodingUTF-16ASCIIKen ThompsonRob PikePlan 9self-synchronizationsurrogate pairs
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.