DGraph 2024 Architecture Upgrade and Performance Optimizations
In 2024 DGraph upgraded its architecture by splitting single clusters into multiple business‑specific clusters, adopting a sharded active‑active topology, and replacing its 1:N thread‑pool with an M:N grouped execution model that uses atomic scheduling, while parallelizing FlatBuffer encoding, streamlining SDK conversions, adding DAG debugging, timeline analysis, and dynamic sub‑graph templates to boost scalability, stability and developer productivity.
DGraph is the next‑generation recommendation engine developed by Dewu, built in C++ and in service since 2021. It provides data pre‑selection and coarse‑ranking for the recommendation pipeline.
Core technical features : multi‑type index layer (KV, KVV, inverted, dense‑KV) with both disk and memory modes, hot‑update mechanisms, and a query layer supporting vector search (IVF & HNSW), KV, inverted, X2I, and graph queries. Java SDK and C++ SDK are exposed.
System dependencies : index lifecycle managed by Dewu Index Platform (DIP), service discovery via ZooKeeper, and container‑based resource scheduling with HPA support.
Service scale : over 100 clusters online, exceeding 1 M QPS during Double‑11 2024.
Architecture upgrade – vertical business‑cluster splitting : the original single‑cluster design faced storage, network, compute, and stability bottlenecks. The upgrade introduces multiple clusters per business line, decoupling resources and enabling horizontal scaling. Access to other DGraph clusters and FeatureStore is added using the existing FlatBuffers protocol.
Distributed capability support : a sharded, active‑active cluster topology (2 shards × 2 nodes × 2 replicas) with symmetric deployment of Proxy and engine on each node. Compatibility is kept for basic KV, inverted, and X2I queries, while DAG queries require explicit proxy operator placement.
Performance optimization – operator execution framework : replaced a 1:N thread‑pool/queue design with an M:N grouped model, reducing lock contention. Scheduler locks were replaced by atomic operations for DAG node state updates, achieving stable latency even under >60% CPU load.
Transmission protocol encoding/decoding optimization : parallelized FlatBuffer block encoding, eliminated redundant Doc→Map→Object conversion in the Java SDK, and streamlined the decode path.
User‑experience improvements : integrated DAG debugging into the DIP platform, logging intermediate operator results keyed by TraceID/DAGID/NodeID. Added Chrome‑based TimeLine analysis for per‑operator timing, and introduced dynamic sub‑graph templates to simplify DAG modifications.
Future outlook : continue to enhance product usability, development and operation efficiency, system stability, scalability, and platformization.
DeWu Technology
A platform for sharing and discussing tech knowledge, guiding you toward the cloud of technology.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.