Common Ciphertext Leakage Scenarios and Escape Testing Methods for File Encryption Software

In the rapidly changing encryption market, endpoint file encryption has become a mainstream focus, with many products competing for market share; improving user experience, product competitiveness, and especially preventing ciphertext cracking and escape are critical concerns for developers.

Ciphertext is defined as data that is encrypted with a reliable algorithm so that anyone without the key and algorithm cannot read it, yet the data must be decrypted to plaintext when executed on a machine, creating a risk that attackers may capture the plaintext.

No encryption technique is unbreakable; the difficulty of cracking depends on design choices, and if confidential handling is not considered from the start, ciphertext leakage becomes inevitable.

To understand ciphertext escape, it is useful to know how drivers control the application layer in Windows. The file system operates through device driver programs, and filter drivers can intercept and modify requests before they reach lower‑level drivers.

The Windows file system hierarchy consists of four layers:

Layer I – can access data read/write by the application itself, allowing only static encryption.

Layer II – can intercept file open/close operations for many files.

Layers III and IV – kernel layers where dynamic encryption/decryption must be implemented.

Based on this understanding, the article presents several common ciphertext leakage scenarios caused by poor design.

Scenario 1: Process Forgery – Encryption formats often embed a process identifier in the file header. If the protected process is not digitally signed, an attacker can rename a benign process (e.g., notepad.exe) to the expected name (e.g., word.exe) and trick the encryption software into decrypting the ciphertext, leading to leakage.

Examples include modifying notepad.exe to masquerade as word.exe, and using the LAN communication tool FeiGe ChuanShu to rename its process and transmit ciphertext without proper protection.

Scenario 2: Driver Hijacking and Disabling – Dynamic encryption can be embedded via hook or filter drivers. Testers can detach the driver (e.g., using tools like IceSword) to stop encryption enforcement, then open the ciphertext at the application layer.

Scenario 3: Temporary File Leakage – Windows caches file data in memory and temporary files (e.g., *.TMP in the %TMP% directory). If encryption software does not encrypt these temporary artifacts, they remain in cleartext and can be recovered. Additionally, prefetch files (*.pf) and pagefile.sys may retain sensitive information.

Scenario 4: File Insertion Objects – Office documents can embed various objects (text, video, other files). Attackers can insert encrypted files as objects within a plain document, then open them with a non‑protected process, causing leakage. Other methods include email exfiltration, memory dumping, pipe redirection, and packaging ciphertext in zip archives.

The article concludes with recommendations for ciphertext escape testing: testers must continuously explore new methods, avoid relying on conventional techniques, and improve their understanding of encryption principles to enhance software quality.

Appendix (from 360 Encyclopedia)

Static Encryption: Encryption performed on a file that is at rest and not actively used, requiring a password, key certificate, or digital signature to decrypt before use.

Dynamic Encryption: Real‑time or transparent encryption applied automatically as files are created, edited, or accessed, making the encrypted file appear as normal to legitimate users while remaining unreadable to attackers.

Admission Signature: A whitelist mechanism that validates a process by its ID before allowing encryption/decryption; unsigned or mismatched processes are blocked.