Douban's Platform Architecture: Online Services, BeansDB, DAE, and DPark

Douban's overall infrastructure is divided into online and offline sections. The online side uses LVS for HA and Nginx as a reverse proxy to achieve load balancing, while the application layer runs on the DAE platform, with typical backend services such as MySQL, memcached, Redis, beanstalkd, and the home‑grown KV store BeansDB.

BeansDB, launched in 2008 and open‑sourced in 2009, started with Tokyo Cabinet and later switched to a Bitcask‑style engine for better performance. It hashes keys to distribute data across nodes with three‑way replication, supports massive storage (hundreds of terabytes), offers simple operations, high performance, availability, easy scaling, and eventual consistency.

The BeansDB proxy is written in Go and provides a simpler design compared to Redis, with a more straightforward value type.

Douban operates two BeansDB clusters: doubandb stores small textual data (reviews, user profiles, posts) to offload MySQL, while doubanfs holds medium‑sized media such as images and audio.

DAE is an internal PaaS built from many existing components, offering simplified security isolation and permission management compared to public clouds; it currently supports Python applications and may later support Go.

The offline side handles data mining and analysis using MooseFS, a C‑based distributed file system similar to HDFS with a well‑implemented FUSE module, and a custom distributed computing platform called DPark.

DPark, originally a Python implementation of Spark, inherits Spark's in‑memory caching to accelerate iterative algorithms, which is crucial for Douban's recommendation workloads. Its functional‑style codebase is concise.

As of March 2014, DPark processes roughly 60–100 TB of data per day, a two‑fold increase over the previous year.

Team

The Douban Platform Department consists of four groups: Core Systems (6 engineers), DAE (4 engineers), DBA (2 engineers), and SA (2 engineers). The department focuses on infrastructure that is not directly visible to users, while product‑line teams handle user‑facing features.

Project initiation depends on whether a service is public or business‑specific; for example, the SMS service started as a product‑line request and later moved to the SA team after becoming widely used.

Core System projects include DPark, BeansDB, MooseFS customizations, search services, long‑connection push services, and about a dozen other initiatives, some of which are now in maintenance mode.

Code review is mandatory across the platform to promote knowledge sharing, given the small team size and many projects.

Operational responsibility lies with the project owner, who must handle incident response, gray‑scale releases, and overall quality assurance.

Without dedicated product managers, engineers are encouraged to identify problems themselves and drive solutions collaboratively.

Douban adopts new technologies aggressively but only after proven, comparable‑scale cases exist and the team fully understands the codebase; consequently, the stack favors Python, C, and Go over Java for critical services.