How a HPE Update Erased 77 TB of Kyoto University Research Data

According to reports, a software update pushed by Hewlett Packard Enterprise (HPE) introduced an infinite loop in a backup script, leading to unexpected behavior that deleted backup data. The incident caused Kyoto University’s supercomputing facilities to lose 77 TB of large‑scale research data.

Kyoto University stated that 34 research groups, encompassing 34 million files, were affected, and nearly one‑third of the groups could not recover their data. The university placed full responsibility on the HPE supercomputing system.

The error stemmed from a script designed to delete log files older than ten days. The update, intended to improve visibility and readability, altered the variable name passed to the find command. However, the modified script was deployed without adequate testing, causing the script to reload during execution, encounter undefined variables, and mistakenly delete original log files stored on the backup disk /LARGE0 instead of only the intended logs.

Initially, the university feared up to 100 TB of data loss, but the final assessment confirmed a loss of 77 TB. The affected backup processes have been paused, and the university plans to resume them after fixing the script, aiming for restoration by the end of the month. It also advises users to back up important files to an alternative system.

Kyoto University, a leading Japanese research institution known for work in chemistry, immunotherapy, and materials science, has not yet identified which specific departments permanently lost data.

HPE publicly accepted "100% responsibility" for the incident in an open letter, acknowledging the oversight in the script’s deployment and the unintended side effects.

Kyoto University's supercomputer cluster consists of three systems: Camphor 2 (5.48 petaflops Cray XC40), Laurel 2 (1.03 petaflops Cray CS400 2820XT), and Cinnamon 2 (42.4 teraflops Cray CS400 4840X).