TDSQL Audit Capability: Architecture, Kafka Integration, and Consistency Hash Implementation

TDSQL provides cloud-based audit capabilities that combine the cost-effectiveness of open-source databases with the security of commercial databases.

The architecture consists of three main modules: Proxy, Kafka, and audit-server.

Proxy: three indistinguishable proxy IPs ensure high availability; audit information is sent to Kafka with compression and requires a majority of nodes to acknowledge before upload; each user instance has a dedicated proxy for concurrent upload to Kafka topics.

Kafka: a distributed publish/subscribe messaging system offering O(1) persistence, high throughput, partitioning, ordered delivery, and support for offline and real-time processing.

audit-server: a distributed service using consistent hashing for routing and multi‑coroutine concurrent processing to achieve second‑level consumption of Kafka data.

Consistent hashing maps keys to a 2^32 bucket space and service nodes to a hash ring; virtual nodes are used to achieve uniform load distribution.

The audit policy includes independent rule loading coroutines, user‑definable priorities, rich rule settings (=, !=, >, >=, <, <=, regex), and two‑factor authentication for data security.

Multi‑coroutine concurrency avoids preemptive scheduling overhead, improving task concurrency.

Fault‑tolerance measures ensure subsystem isolation, prevent data loss by recording offsets, require majority Kafka acknowledgments for audit data, provide second‑level failure detection for Kafka/MongoDB, support auto‑scaling via Tencent Cloud MongoDB, and guarantee message ordering via timestamps.

Tencent Cloud MongoDB is used as the storage backend, offering cloud storage, high performance/reliability as replica sets or sharded clusters, integrated backup and scaling, and user‑initiated audit enabling.

Benefits of using Tencent Cloud MongoDB include online dual‑copy safety, backup‑based disaster recovery, high‑performance SSD‑based servers, transparent scaling/migration, and 24/7 professional monitoring.