MySQL's utf8 Isn't Real UTF-8: Why You Must Use utf8mb4

When inserting an emoji such as 😓 into a MySQL column defined with the utf8 character set, the statement fails with:

[Err] 1366 - Incorrect string value: '\xF0\x9F\x98\x93' for column 'NAME' at row 1

The error occurs because the emoji requires four bytes, but MySQL's utf8 only allows a maximum of three bytes per character. Changing the database, table, and column character set to utf8mb4 resolves the issue, as shown by the successful insert of '陈哈哈😓😓'.

1. utf8mb4 Is the Real UTF-8

MySQL's utf8 is a proprietary encoding that supports only a subset of Unicode characters (max three bytes). True UTF-8 (RFC 3629) supports up to four bytes per character. In utf8, Chinese characters occupy three bytes, while ASCII characters use one byte. Emoji, some complex characters, and certain traditional Chinese characters need four bytes, so they cannot be stored in utf8 columns. The article emphasizes that all MySQL and MariaDB users should migrate to utf8mb4 and never use utf8 again.

2. History of MySQL's utf8

MySQL added UTF-8 support in version 4.1 (2003), based on the older RFC 2279 which allowed up to six bytes per character. On 28 March 2002, the first MySQL 4.1 preview used RFC 2279. In September 2002, a source-code change limited UTF-8 to three-byte sequences. The commit author and rationale were lost during the later migration from BitKeeper to Git, and the September 2003 mailing list offers no explanation.

The author speculates that MySQL developers wanted to optimize CHAR columns: if every row uses a fixed byte length, MySQL can gain performance. With six bytes per character, CHAR(1) would consume six bytes, CHAR(2) twelve bytes, etc. This approach was never released, but the documentation described it, leading users to believe utf8 supported six bytes. Developers then feared users would define CHAR columns with utf8 expecting fixed-width storage, so they capped the encoding at three bytes to keep CHAR sizes predictable. The result: users seeking space/speed trade-offs got larger, slower columns, while users needing correctness could not store four-byte characters.

Because fixing the broken utf8 would require all users to rebuild their databases, MySQL introduced utf8mb4 in 2010 to provide genuine UTF-8 support without breaking existing installations.

Background: Encoding and UTF-8

Computers store text as binary. For example, the character "C" is stored as 01000011 (decimal 67). The process: (1) read bits → number 67; (2) look up 67 in the Unicode character set → "C". Conversely, when saving "C", the computer maps it to Unicode code point 67, then encodes 67 as 01000011. Nearly all network applications use the Unicode character set. UTF-32 uses 32 bits per character (simple but wasteful). UTF-8 saves space: ASCII characters use 8 bits, less common characters use 16, 24, or 32 bits. An article like this one occupies roughly one-quarter the space in UTF-8 compared to UTF-32.

Summary

Almost every online article mistakenly treats MySQL's utf8 as true UTF-8. When provisioning MySQL or MariaDB, always set the relevant character sets to utf8mb4. Future maintainers will silently appreciate the foresight.