ByteGraph: ByteDance’s Distributed Graph Database and Graph Computing System – Architecture, Data Model, and Practices

ByteDance’s products generate massive social‑graph data, which can be abstracted into three categories—user information and relationships, content items, and user‑content interactions—forming a large directed‑property graph that drives recommendation and user experience.

To serve online OLTP workloads, ByteDance built ByteGraph, a distributed graph storage system that supports a directed‑property graph model, Gremlin queries, and can scale to tens of millions of QPS with millisecond latency across all product lines.

The graph data model consists of vertices (points) and edges, each identified by a uint64_t id and a uint32_t type , with arbitrary KV attributes. Edges are directional and can be forward, reverse, or bidirectional.

- Vertex id (uint64_t): e.g., user id
- Vertex type (uint32_t): e.g., app ID
- Vertex attributes: KV pairs such as name, age, gender
- Edge connects two vertices, has a type string (e.g., "follow"), a timestamp (uint64_t), and KV attributes
- Edge direction: forward (A → B), reverse (B ← A), bidirectional (A ↔ B)

Typical Gremlin‑style usage examples include creating a follow relationship, querying two‑hop neighbors, and retrieving friends‑of‑friends, illustrated with concise code snippets.

// Create vertices A and B
g.addV().property("type", A.type).property("id", A.id)
g.addV().property("type", B.type).property("id", B.id)
// Create a follow edge A → B
g.addE("关注").from(A.id, A.type).to(B.id, B.type).property("ts_us", now)

ByteGraph’s architecture mirrors a classic three‑layer database stack: a query layer (bgdb) written in Go that parses Gremlin, plans execution, and routes requests; a storage/transaction layer (bgkv) implemented in C++ that provides high‑performance point‑and‑edge reads/writes with operator push‑down and caching; and a disk‑storage layer built on ByteDance’s proprietary distributed KV store.

Because graph vertices can have billions of edges (e.g., a celebrity with tens of millions of followers), ByteGraph adopts a hybrid storage scheme: small out‑degree vertices use a single KV entry (level‑1 storage), while high‑degree vertices are split into multiple KV entries organized as a logical distributed B‑Tree (level‑2 storage). This keeps KV value sizes uniform and enables efficient range queries.

Hotspot read/write traffic is mitigated by multi‑level query caching in bgdb and a group‑commit mechanism in bgkv that batches writes to the underlying KV store, ensuring stable performance even under extreme concurrency.

Indexing is performed primarily on edge timestamps, with optional secondary indexes on target vertices or custom attributes to accelerate common queries such as recent likes or time‑windowed friend additions.

Beyond storage, ByteDance developed a graph‑computing subsystem to handle large‑scale OLAP workloads. The article surveys computation models (vertex‑centric Pregel, GAS in PowerGraph), graph partitioning strategies (edge‑cut vs. vertex‑cut), execution models (synchronous, asynchronous, semi‑synchronous), and communication models (push, pull, shared memory). It also reviews existing open‑source systems (Pregel/Giraph, GraphX, Gemini) and describes ByteDance’s custom extensions to Gemini (Titanic‑scale vertex ID encoding, hierarchical chunk‑based partitioning, adaptive push/pull).

Future work includes moving from pure in‑memory to hybrid storage with emerging hardware (AEP, NVMe), supporting dynamic graph updates, exploring heterogeneous acceleration (GPU/FPGA), and designing a high‑level graph query language to decouple business logic from engine internals.

In summary, ByteGraph demonstrates how a large internet company can design a high‑performance, scalable graph database and compute platform that serves both OLTP and OLAP needs for billions of users and trillions of edges.