Implementing a Transparent Data Encryption Framework for Database Security

This article discusses the implementation of a transparent data encryption framework for database security in internet companies. The author, a senior developer at Tencent, addresses the critical need for data security, particularly for sensitive information such as ID numbers, phone numbers, and card numbers.

The article begins by examining various encryption implementation schemes and their pros and cons, ultimately concluding that database driver layer modification offers the least drawbacks. It then explores the concept of database access architecture and how adding an intermediate layer can solve many problems.

The core solution proposed is using Aspect-Oriented Programming (AOP) to intercept and rewrite business SQL statements. The author provides detailed pseudo-code demonstrating how to implement an XDriverInterceptor that can transparently encrypt and decrypt data without modifying business logic or SQL.

The article delves into SQL parsing, explaining the process of lexical and syntactic analysis to generate abstract syntax trees (AST). It covers how to rewrite SQL statements by transforming the AST and then converting it back to SQL with encrypted column names.

Performance considerations are addressed through concurrency control using segmented locks and caching strategies. The article explains different caching policies for parameterized queries versus string-concatenated SQL.

Configuration management is discussed using SPI (Service Provider Interface) mechanisms, allowing for custom encryption algorithms and key generation methods. The framework supports both built-in implementations and user-defined extensions through YAML configuration files.

The article also covers handling existing data encryption scenarios, providing a three-phase migration strategy: pre-migration (dual-write with plaintext queries), migration (dual-write with encrypted queries), and post-migration (single-write with encrypted queries only).

Finally, the article summarizes how the framework addresses key requirements: minimal code intrusion, low adoption cost, broad framework coverage, high performance and availability, and support for existing data encryption.