Why Does Python 2 Encoding Fail with Chinese Characters? A Step‑by‑Step Guide

Previously we introduced the theory of Unicode and UTF‑8 encoding; you can read that article here .

We then demonstrate string encoding in a Python 2 environment on both Windows and Linux to deepen understanding.

1. Windows demonstration

In the Windows command line, entering the strings abc and u'abc' shows that the former is a byte string while the latter is Unicode. Encoding both with utf‑8 works without error.

When using Chinese characters, the byte string raises an error while the Unicode string does not. This occurs because the byte string is encoded as GB2312 on Windows, whereas the Unicode string is already decoded. To convert the GB2312 string to UTF‑8, you must first decode it to Unicode and then encode it to UTF‑8.

2. Linux demonstration

Repeating the same steps on Linux yields the same final result, but the process differs because Linux uses UTF‑8 as the default encoding. Directly encoding a Chinese byte string to UTF‑8 fails; you must decode it using UTF‑8 first, then re‑encode.

The key point is that str1.encode('utf‑8') appears to work only if str1 is already Unicode. In Python 2, encode() first decodes the string using the system’s default codec, which is ASCII on many systems, leading to a decoding error before the UTF‑8 encoding step.

Thus, attempting to encode a Chinese string with ASCII decoding causes an error, and the subsequent encode('utf‑8') never executes.

We conclude the discussion of Python 2 string encoding here and will cover Python 3 encoding issues in the next article.