Why MySQL’s utf8 Is Not Real UTF‑8 and How to Avoid Garbled Text

1. Character Set and Encoding

Two concepts are clarified: a character set is the collection of characters (e.g., ASCII has 128 characters, Unicode covers almost all languages and emojis), while an encoding defines how those characters are mapped to binary for storage. In everyday language the two terms are often mixed, and MySQL names such as utf8mb4 actually refer to a combination of character set and encoding. The essential rule is that the same encoding used for storing must be used for reading; otherwise garbled text appears.

UTF‑8 is a variable‑length encoding where a character occupies 1–4 bytes: ASCII uses 1 byte, common Chinese characters 3 bytes, and emojis or rare characters 4 bytes. The 4‑byte case is the key to the next pitfall.

2. MySQL’s utf8 Is a Pitfall

MySQL’s utf8 is not true UTF‑8; it is an alias for utf8mb3, which limits each character to a maximum of 3 bytes. Consequently, emojis and some rare Chinese characters cannot be stored: attempts either raise Incorrect string value errors or truncate the data into garbage.

The true UTF‑8 in MySQL is utf8mb4, a superset that can store all Unicode characters, including emojis.

-- ✗ Cannot store emoji, may error or truncate
CREATE TABLE t1 (name VARCHAR(50)) CHARSET=utf8;
-- ✓ Use utf8mb4, everything can be stored
CREATE TABLE t2 (name VARCHAR(50)) CHARSET=utf8mb4;

Conclusion: always use utf8mb4 and never rely on utf8. MySQL has deprecated utf8mb3 and will eventually make utf8 an alias of utf8mb4.

This choice also affects index length: because utf8mb4 can use up to 4 bytes per character, a VARCHAR(50) column may occupy up to 201 bytes (200 bytes for data plus 1 byte length marker), influencing key_len calculations.

3. Collation (Comparison Rules)

While the character set decides how data is stored, the collation decides how strings are compared, sorted, and matched (e.g., the behavior of = and LIKE).

Collation suffixes have clear meanings: _ci: case‑insensitive (e.g., 'A' = 'a') _cs: case‑sensitive _bin: binary comparison, all differences matter _ai: accent‑insensitive (e.g., 'e' = 'é') _as: accent‑sensitive

Common collations for utf8mb4: utf8mb4_general_ci: fast but less precise ordering utf8mb4_unicode_ci: Unicode‑based, more accurate, slightly slower utf8mb4_0900_ai_ci: MySQL 8.0 default, based on Unicode 9.0, case‑ and accent‑insensitive utf8mb4_bin: binary, strict case‑sensitive comparison

Practical trap (MySQL 8.0 default): utf8mb4_0900_ai_ci treats 'a', 'A', 'å', 'ä' as equal. A UNIQUE column may reject 'café' as a duplicate of 'cafe'. Use utf8mb4_bin or collations with _as / _cs when strict distinction is required.

4. Four Levels of Character Set Configuration

MySQL applies character set and collation at four hierarchical levels, from most general to most specific, with the nearest level overriding higher ones:

Server → Database → Table → Column
(least specific)          (most specific)

Setting utf8mb4 at the server or database level ensures that new tables and columns inherit it automatically. Existing tables that still use utf8 can be converted with:

-- Convert an entire table to utf8mb4 (including all columns)
ALTER TABLE t CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

5. How Garbled Text Happens

Garbling occurs when the encoding used for storing (the column’s charset) differs from the encoding used for reading (client, connection, or result). MySQL controls this with three system variables: character_set_client: what the server assumes the client’s SQL is encoded in character_set_connection: the charset the server converts the SQL to for processing character_set_results: the charset used for sending results back to the client

Typical conversion flow:

Client sends SQL (character_set_client)
 → Server converts to character_set_connection
 → Data stored using column charset
 → Results sent using character_set_results

If any step mismatches, the stored bytes are interpreted incorrectly, producing garbled output. Classic example: the client sends UTF‑8 data but character_set_client is set to latin1, so the server misinterprets the bytes.

One‑line fix:

SET NAMES utf8mb4;  -- sets client, connection, and results to utf8mb4

In practice this is usually configured in the connection string, e.g. for JDBC:

jdbc:mysql://host:3306/db?useUnicode=true&characterEncoding=UTF-8

Note: the JDBC characterEncoding parameter expects the Java charset name (UTF‑8), not MySQL’s utf8mb4; modern drivers map it automatically.

Debugging tip: trace the chain “client encoding → connection charset → column charset” and ensure they match.

6. Version Differences and Practical Recommendations

MySQL 5.7 defaults to latin1 and latin1_swedish_ci; MySQL 8.0 defaults to utf8mb4 and utf8mb4_0900_ai_ci. Therefore, on 5.7 you must explicitly specify utf8mb4 for databases, tables, and columns; 8.0 does this automatically but you still need to watch the default collation’s accent‑insensitive behavior.

Recommendations:

Use utf8mb4 everywhere (server, database, table, column) and avoid utf8 / utf8mb3 and latin1.

Set the desired collation when creating a database so tables and columns inherit it.

Specify characterEncoding=UTF-8 (or execute SET NAMES utf8mb4) in connection strings.

Choose collations based on needs: the default ci series for most cases; utf8mb4_bin or collations with _as / _cs when case or accent must be distinguished.

When migrating old schemas, convert tables with ALTER TABLE … CONVERT TO CHARACTER SET utf8mb4 and verify that the new default collation does not introduce unexpected UNIQUE key conflicts.

Bottom line: store data with utf8mb4, and when garbled text appears, follow the client → connection → column chain to locate the mismatched encoding.