Why Python’s Unicode Integration Still Breaks File Names and How to Fix It

Unicode has solved many problems, but legacy encodings such as ISO‑8859* and CP437 still cause chaos, especially for non‑Western languages, and today work is being done on encoding emojis.

Beyond emojis, Python 3’s full Unicode integration brings new pain points. Converting a podcast title with smart quotes to ASCII triggers errors in the open‑source gPodder client, leading to tracebacks. The pexpect documentation suggests using logfile = sys.stdout to display terminal interaction, which in this case caused an error.

Filename handling is especially problematic. The author processed data from 20 years ago, before UTF‑8 became the standard for filenames. Those legacy filenames still work on UNIX and can be archived with tar, but storing them as strings results in encoding errors. The correct approach is to treat filenames as bytes, not str, although Python code often fails to do so.

Working with mtree‑generated data that uses octal escapes for special characters revealed many incorrect solutions, including regex‑based fixes that do not work.

An undocumented function, codecs.escape_decode, can correctly decode such filenames; the author ultimately used it (see image below).

Additionally, avoid writing if filetype=="file" because the comparison evaluates to False when the value is a bytes object ( b"file").

To correctly handle Unix filenames in Python you must:

Use a processing path that completely avoids Python str handling for filenames.

Use sys.{stdin,stdout}.buffer instead of the plain sys.stdin/stdout streams.

Pass filenames as bytes to all relevant functions. See PEP 471: the os module functions accept either bytes or str, but str is recommended for Unicode support; note that Windows has not supported bytes filenames since Python 3.3.

Update: If you need to receive filenames on the command line, ensure the environment correctly handles Unicode and byte strings.