Why MySQL 8.0.15 Leaks Internal Temp Files and How Upgrading to 8.0.16 Fixes It

Background

On September 12, 2016 MySQL 8.0 was released, introducing the MySQL Group Replication (MGR) feature as a built‑in high‑availability solution with simple configuration, supporting both single‑primary and multi‑primary modes.

Problem Observed

After several months of stable operation, alerts appeared indicating three symptoms:

Program error

ERROR   dbtools.get_table_mysql_connect:865  (1135, "Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug")

Database error

1135, "Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug"

OS‑level abnormality

MySQL opened roughly 120,000 internal temporary‑table files.

First Analysis

The usual cause of similar symptoms is a tmpdir set to /tmp leading to root‑partition exhaustion, but disk space was sufficient. The issue was traced to internal temporary tables not being released, resulting in an abnormal accumulation of files (over 120,000).

Official documentation states that starting with MySQL 8.0.16 the storage engine for on‑disk internal temporary tables is always InnoDB and the variable internal_tmp_disk_storage_engine was removed.

Storage Engine for On-Disk Internal Temporary Tables
Starting with MySQL 8.0.16, the server always uses the InnoDB storage engine for managing internal temporary tables on disk.
In MySQL 8.0.15 and earlier, the internal_tmp_disk_storage_engine variable was used to define the storage engine used for on-disk internal temporary tables. This variable was removed in MySQL 8.0.16, and the storage engine used for this purpose is no longer user-configurable.

This indicated that the internal_tmp_disk_storage_engine parameter, which could be set to MyISAM in earlier versions, was no longer configurable in 8.0.16.

First Fix Attempt

We changed the parameter back to the pre‑5.7.5 default (MyISAM) and restarted the cluster. During maintenance we observed that internal temporary files persisted after the client stopped, only being removed when mysqld stopped.

The next day the file count remained high:

[root@localhost] lsof -u mysql|wc -l
21323

At this rate the number of open files would reach 120,000 within a week, reproducing the original failure.

Second Analysis

Searches on Baidu yielded little, but an Oracle Community post titled “Mysql daemon not releasing deleted temp file” matched our symptoms and identified a known bug (Oracle BUG 28039829) affecting MySQL 5.7.22‑5.7.25 and 8.0.14‑8.0.15, fixed in 5.7.26 and 8.0.16.

you are hitting a known bug that affected MySQL 5.7.22 - 5.7.25 and 8.0.14 - 8.0.15.  It has been fixed in 5.7.26 and 8.0.16.  Oracle BUG id is 28039829 "File handles are leaked for SELECT queries involving complex joins."  The way MySQL uses on-disk internal temp tables (those produced during a query, not the explicit temporary tables that a user can create), to not leave any temp files behind in case of a server crash, by design MySQL creates the tmp file, and then immediately unlinks it (so it appears to the OS as deleted), but MySQL keeps the descriptor open to the file.

Second Fix Attempt

We upgraded the cluster from 8.0.15 MGR to an 8.0.16 Master‑Slave architecture to verify whether the bug was version‑specific. After one week no recurrence was observed, confirming that 8.0.16 alone resolves the issue.

Subsequently we switched the 8.0.16 Master‑Slave setup back to an 8.0.16 MGR cluster. After another week the problem did not reappear, demonstrating that the bug is unrelated to the MGR deployment mode.

Conclusion

The leak of internal temporary files is a known bug in MySQL 8.0.15 (BUG 28039829). Upgrading to MySQL 8.0.16 eliminates the issue regardless of whether the deployment uses MGR or classic master‑slave replication. This case provides valuable reference for production environments in China that still run MySQL 8.0.15.