A Programmer’s Intro to Unicode

Diversity and Inherent Complexity

Unicode aims to represent every written system. The Unicode Consortium’s goal is “to enable people everywhere to use any language on computers.” Unicode currently supports 135 scripts covering about 1,100 languages, with more than 100 scripts still awaiting inclusion.

Programming languages provide libraries that hide low‑level details, but developers must understand key Unicode features to know when and how to apply them.

Unicode Code Space

A Unicode code point is identified by a hexadecimal number prefixed with “U+” (e.g., U+0041 for “A”, U+03B8 for “θ”). The full set of possible code points is the codespace , containing 1,114,112 points. Only 128,237 points (~12%) are assigned; 137,468 points are reserved for the private‑use area.

The codespace is visualised as 17 planes, each with 65,536 points. Plane 0 (the Basic Multilingual Plane, BMP) holds almost all characters needed for modern text. Plane 1 contains historic scripts, emojis and other symbols. Plane 2 contains many less‑common and historic Han characters. Planes 14‑16 are largely empty; planes 15‑16 are reserved for private use.

Heat‑maps based on Wikipedia and Twitter samples show that the vast majority of real‑world text resides in the BMP, with only sparse usage in planes 1‑2 except for emojis, which appear as bright spots in plane 1.

Encoding Forms

Code points must be represented as bytes in memory or files. Storing each point as a 32‑bit integer (UTF‑32) is simple but wasteful, using four bytes per character.

Variable‑length encodings UTF‑8 and UTF‑16 are more common and length‑efficient for low‑value code points.

UTF‑8

UTF‑8 encodes a code point using 1 to 4 bytes. The leading bits of each byte indicate whether it is a single‑byte character, the start of a multi‑byte sequence, or a continuation byte.

UTF‑8 (binary)                Code point (binary)                Range
0xxxxxxx                      xxxxxxx                           U+0000–U+007F
110xxxxx 10yyyyyy             xxxxxyyyyyy                      U+0080–U+07FF
1110xxxx 10yyyyyy 10zzzzzz    xxxxyyyyyyzzzzzz                 U+0800–U+FFFF
11110xxx 10yyyyyy 10zzzzzz 10wwwwww  xxxyyyyyyzzzzzzwwwwww   U+10000–U+10FFFF

UTF‑8 is ASCII‑compatible: bytes 0‑127 encode the same characters as ASCII, and ASCII control bytes never appear in multi‑byte sequences. This allows existing ASCII files to be interpreted as UTF‑8 and permits common delimiters (null, newline, tab, comma, slash) to work unchanged.

Processing UTF‑8 requires decoding to obtain code points or grapheme clusters, and length calculations must decide whether to count bytes, code points, or rendered width.

UTF‑16

UTF‑16 uses 16‑bit units. Code points in the BMP are stored as a single unit; those above U+FFFF are encoded as a surrogate pair (two 16‑bit units). The high‑surrogate range is U+D800–U+DBFF and the low‑surrogate range is U+DC00–U+DFFF. Surrogate code points are illegal on their own and never appear in UTF‑8 or UTF‑32.

UTF‑16 (binary)                Code point (binary)                Range
xxxxxxxxxxxxxxxx              xxxxxxxxxxxxxxxx                  U+0000–U+FFFF
110110xxxxxxxxxx 110111yyyyyyyyyy  xxxxxxxxxxyyyyyyyyyy + 0x10000   U+10000–U+10FFFF

JavaScript strings and the Windows Win32 API use UTF‑16. Historically Windows lacked UTF‑8 support, but Windows 10 version 1903 added it.

Combining Marks

Unicode supports dynamic composition: a base character followed by one or more combining marks to produce a single visual character. Example: “Á” can be represented as U+0041 ("A") plus U+0301 (combining acute accent). Pre‑composed characters such as U+00C1 also exist for common combinations.

Combining marks enable arbitrary stacking, as seen in “Zalgo” text, and are used in Arabic/Hebrew vowel notation (niqqud), Devanagari vowel signs, and Korean jamo composition.

Canonical Equivalence

Multiple code‑point sequences can represent the same perceived character. “Á” can be encoded as the single pre‑composed code point U+00C1 or as the two‑code‑point sequence U+0041 U+0301. Vietnamese “ệ” can be expressed in five different ways, ranging from fully pre‑composed to fully decomposed sequences.

These sequences are called canonically equivalent . Applications that search, sort, or render text should treat canonically equivalent strings as identical.

Normalization Forms

Unicode defines several normalization forms to transform strings into a consistent representation:

NFD fully decomposes characters into base characters and combining marks, also sorting marks by rendering position.

NFC composes characters where possible, leaving remaining marks decomposed.

NFKD and NFKC perform compatibility decomposition, handling characters that are visually similar but not identical.

Grapheme Clusters

A grapheme cluster is a sequence of one or more code points that a user perceives as a single character. UAX #29 defines the precise rules, covering base characters plus combining marks, Korean jamo, and emoji ZWJ sequences.

Grapheme clusters are essential for text editing: cursor movement, selection, and length limits should operate on clusters rather than raw code points or bytes to avoid breaking combined characters.

References

Unicode Standard – https://www.unicode.org/versions/Unicode17.0.0/

UTF‑8 Everywhere – https://utf8everywhere.org/

"The Dark Corners of Unicode" by Eevee – https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/

ICU (International Components for Unicode) – https://icu.unicode.org/

Python 3 Unicode How‑to – https://docs.python.org/3/howto/unicode.html

Google Noto Fonts – https://fonts.google.com/noto

UAX #29 (Unicode Text Segmentation) – https://www.unicode.org/reports/tr29/

Emoji ZWJ sequences – https://blog.emojipedia.org/emoji-zwj-sequences-three-letters-many-possibilities/