Why Data Loss Happens: Hidden CPU Silent Errors and How to Prevent Them

Background

For data storage systems, guaranteeing that data is neither lost nor corrupted is the baseline requirement and the most difficult part; losing a 10‑day‑old data center can cause 93% of companies to go bankrupt within a year.

Definitions

Data not lost means the content (file or metadata) exists. Data not corrupted means the content exists but may contain errors (e.g., bit flips). Data consistency is a stricter requirement and is not covered in depth.

Common Bit‑Flip Sources

Bit flips can occur in disks, memory, network, and CPU.

Disk: flips may happen in the storage medium or during read; detection uses extra checksum fields such as the Data Integrity Field (e.g., 520‑byte blocks) and SMART counters (UltraDMA CRC Error Count, Soft ECC Correction, Hardware ECC Recovered).

Memory: susceptible to interference (crosstalk, cosmic rays); mitigated with ECC memory.

Network: errors in cables, interfaces, or NIC components; mitigated with checksums.

CPU: silent data errors (SDE) that are not reported to the OS, causing incorrect computation results without alerts.

CPU Silent Data Error (SDE) Investigation

Two core modules on the same server reported checksum anomalies. After eliminating software causes, the team repeatedly computed MD5 on known data in /dev/shm and compared it with the expected digest. Occasionally the CPU returned an incorrect MD5 while the OS remained stable.

$ pwd
/dev/shm
$ cat t.py
import os
import sys
import hashlib

data = open("./data").read()
hl = hashlib.md5()
hl.update(data)
digest = hl.hexdigest()
print "digest is %s" % digest
if digest != "a75bca176bb398909c8a25b9cd4f61ea":
    print "error detected"
    sys.exit(-1)

The CPU vendor confirmed a hardware defect in one core and offered short‑term monitoring tools and a long‑term detection solution.

Design Considerations for “Data Not Lost & Not Corrupted”

A systematic approach is needed to protect against hardware errors and software bugs. Key dimensions include:

Data redundancy (replication, erasure coding) to enable repair.

Backup strategies (incremental, full, snapshots) for recovery.

End‑to‑end CRC checks across compute, network, and storage layers.

Comprehensive logging of front‑end and back‑end data changes.

Detection mechanisms for incremental, stored, and full‑scale data (log‑based checks, periodic scans).

Prioritizing metadata detection because of its higher impact.

Fault Modes and Detection

Faults are classified as hardware or software errors. Detection methods include XOR, CRC, LDPC, and application‑level checks. Repair methods involve redundancy‑based reconstruction or backup restoration.

Software Bug Impact

Software bugs can cause data loss if no redundancy or backup exists. Detecting such bugs requires logging of incremental, stored, and full data changes and decoupled error‑detection modules.

Conclusion

Ensuring data is neither lost nor corrupted requires a holistic design covering redundancy, backup, error detection, and recovery. While current mechanisms mitigate many hardware errors, CPU silent data errors remain a challenging area that demands both technical and managerial diligence.