MySQL's utf8 Is Not True UTF-8: Why You Need utf8mb4

Error Recap

Inserting emoji directly into a SQL statement with a utf8 column fails:

INSERT INTO `csjdemo`.`student` (`ID`, `NAME`, `SEX`, `AGE`, `CLASS`, `GRADE`, `HOBBY`) VALUES ('20', '陈哈哈😓', '男', '20', '181班', '9年级', '看片儿');

[Err] 1366 - Incorrect string value: '\xF0\x9F\x98\x93' for column 'NAME' at row 1

Changing the database, system, and column encoding to utf8mb4 resolves the issue:

INSERT INTO `student` (`ID`, `NAME`, `SEX`, `AGE`, `CLASS`, `GRADE`, `HOBBY`) VALUES (null, '陈哈哈😓😓', '男', '20', '181班', '9年级', '看片儿');

MySQL's utf8 Quirk

MySQL's “utf8” is not true UTF-8. In MySQL, “utf8” supports a maximum of three bytes per character, while true UTF-8 supports up to four bytes.

In utf8, Chinese characters occupy three bytes; digits, English letters, and symbols occupy one byte. However, emoji and some complex or traditional characters require four bytes, causing insertion failures. The fix is to use utf8mb4.

MySQL never fixed this bug. In 2010 they released the “utf8mb4” character set to work around the problem, but they did not widely publicize it (perhaps due to embarrassment). Consequently, many online resources still incorrectly recommend “utf8”.

1. utf8mb4 Is True UTF-8

Yes, MySQL's utf8mb4 is the genuine “UTF-8”. MySQL's utf8 is a proprietary encoding that cannot represent many Unicode characters.

All MySQL and MariaDB users currently using “utf8” should switch to “utf8mb4” and never use “utf8” again.

What Is Encoding? What Is UTF-8?

Computers store text as 0s and 1s. For example, character “C” is stored as “01000011”. Display involves two steps:

The computer reads “01000011”, gets the number 67 (since 67 is encoded as “01000011”).

The computer looks up 67 in the Unicode character set and finds “C”.

Conversely, to send “C”: the computer maps “C” to Unicode code point 67, encodes 67 as “01000011”, and transmits it.

Almost all network applications use the Unicode character set. Unicode contains millions of characters. The simplest encoding is UTF-32 (32 bits per character), but it wastes space. UTF-8 saves space: “C” uses 8 bits, while rare characters like “😓” use 32 bits; others use 16 or 24 bits. An article like this encoded in UTF-8 occupies roughly one‑quarter the space of UTF-32.

2. Brief History of utf8

Why did MySQL make “utf8” non‑compliant? Clues appear in version history.

MySQL added UTF‑8 support in version 4.1 (2003), but the modern UTF‑8 standard ( RFC 3629 ) came later. The older standard ( RFC 2279 ) allowed up to six bytes per character. On March 28, 2002, MySQL developers used RFC 2279 in the first MySQL 4.1 preview.

In September 2002, they changed the source code: “UTF8 now only supports 3‑byte sequences.” The committer’s name and rationale were lost during migration from BitKeeper to Git, and the September 2003 mailing list offers no explanation.

We can speculate: In 2002, MySQL decided that if every row used a fixed byte length, performance could improve significantly. Users would define text columns as CHAR, each CHAR column having a fixed character count; shorter values are space‑padded, longer values truncated.

Initially, MySQL tried UTF‑8 with six bytes per character: CHAR(1) used 6 bytes, CHAR(2) used 12 bytes, etc. This was the correct behavior, but that version was never released. However, the documentation stated this, and it spread widely.

MySQL developers or vendors likely feared users would:

Define columns as CHAR (which was faster in MySQL at the time, though not after 2005).

Set the CHAR column's encoding to “utf8”.

The guess is that developers wanted to help users win on both space and speed, but they broke “utf8” encoding.

Result: no winners. Users seeking space/speed gains actually used more space and got slower performance with “utf8” CHAR columns. Users needing correctness could not store four‑byte characters like “😓”.

Once this non‑compliant character set shipped, MySQL could not fix it without forcing all users to rebuild their databases. Finally, in 2010, MySQL re‑released “utf8mb4” to support true UTF‑8.

Summary

Almost every article online treats “utf8” as true UTF‑8, including my earlier articles and projects. I hope more developers see this.

Many are in the same boat.

When setting up MySQL or MariaDB databases in the future, remember to change all relevant encodings to utf8mb4 . Someday, the programmer who takes over or your boss will silently appreciate your technical prowess.