Why C# Char '\x1d11e' Is Invalid and How Code Pages Influence Encoding

Character Encoding Development Questions

1. Why is char c='\x1d11e' illegal in C#? C# supports Unicode only in the Basic Multilingual Plane because the char type is 2 bytes. The literal \x1d11e refers to a code point outside this plane.

2. How do programming languages and operating systems display characters from code points? Font files contain mappings from Unicode code points to glyphs, which are used to render characters.

3. What is a “code page” and is it Windows‑only? A code page (e.g., CP936, CP437) originated in MS‑DOS 3.3 as an internal code table mapping byte values to characters. Both Windows and Linux have code page concepts, though the numeric identifiers differ.

Common Windows code pages include:

932 – Japanese

936 – Simplified Chinese (GBK)

949 – Korean

950 – Traditional Chinese (Big5)

437 – Original IBM PC extended ASCII

1200 – UCS‑2LE (Unicode little‑endian)

1201 – UCS‑2BE (Unicode big‑endian)

65000 – UTF‑7

65001 – UTF‑8

Code pages are divided into ANSI and OEM; OEM resides in BIOS, while ANSI is at the software layer. CP936 belongs to both categories. Console windows use OEM code pages, other contexts use ANSI.

4. What are U+00A0 / C2A0 and why do they cause space garbling? U+00A0 is a NO‑BREAK SPACE used in Unicode typography. Its UTF‑8 bytes are C2 A0. Because GBK lacks this character, converting a UTF‑8 file containing it to GBK produces garbled spaces.

When copying text containing   from a browser into applications like OneNote and then into a GBK‑encoded file, the space may become U+00A0 and appear as garbage.

Two ways to fix the issue:

Replace   with U+00A0 before copying to the clipboard.

Replace   with a regular space, place it inside a pre block, then copy.

Experiments show default Visual Studio settings encode C++ source files using the active code page (e.g., CP936) and C# source files as UTF‑8, while the compiled binaries use UTF‑16 (little‑endian) for C# and UTF‑8 for Java class files.

Additional questions:

1. How to determine a text’s encoding? It cannot be guaranteed 100%; Windows provides the IsTextUnicode API to guess. Other methods include checking file headers or using statistical detection libraries such as uchardet .

2. Why does garbled text appear and how to avoid it? Garbling results from incorrect decoding. Using the correct encoding throughout reduces the risk, though cross‑system differences make it hard to eliminate entirely.

3. How to prevent Linux/Windows encoding mismatches? Prefer BOM‑less UTF‑8 for source files.

4. How are different encodings converted? Conversion tables (code pages) map Unicode to the target encoding; installing a code page installs its conversion table.