Inside Douban’s Scalable Architecture: BeansDB, DAE, and DPark Explained

This article, based on an interview with Douban engineer Hong Qiangning, outlines Douban’s overall platform architecture, key components, and team experience.

Architecture

Douban’s infrastructure is divided into online and offline parts. The online side uses LVS for HA and Nginx as a reverse proxy for load balancing. Application servers run on the DAE platform, and supporting services include MySQL, Memcached, Redis, Beanstalkd, and the home‑grown KV store BeansDB.

BeansDB

BeansDB, an open‑source KV database inspired by AWS DynamoDB, started in 2008 and was open‑sourced in 2009. It originally used Tokyo Cabinet, later switched to the Bitcask storage format for better performance. Data is sharded by hashing keys across nodes with replication (three copies written, one read). BeansDB supports massive data volumes (hundreds of terabytes) and offers simple operations, high availability, easy scaling, and eventual consistency.

The BeansDB proxy is written in Go. Two clusters are deployed: doubandb for small text data (reviews, user profiles) to offload MySQL, and doubanfs for medium‑size media files (images, audio).

DAE Platform

DAE is an internal PaaS built on existing components, simplifying security, isolation, and permission management compared with public clouds. Currently it supports Python applications and is planned to be open‑sourced; future support for Go is considered.

Offline Processing

The offline side handles data mining and analysis using MooseFS, a distributed file system similar to HDFS written in C, and DPark, a Python‑based distributed computing platform derived from Spark.

DPark inherits Spark’s in‑memory caching to accelerate iterative algorithms, which is crucial for Douban’s recommendation calculations. Its functional‑style API makes code concise.

As of March 2014, DPark clusters process 60–100 TB of data daily, roughly double the volume of the previous year.

Team Structure

The Douban Platform Department consists of four groups: Core System (6 engineers), DAE (4 engineers), DBA (2 engineers), and SA (2 engineers). Projects are categorized as public (platform) or business‑specific, with public projects originating from product lines when they become widely used.

Core System projects include BeansDB, DPark, MooseFS, search services, and long‑connection push services. Code review is mandatory to promote knowledge sharing, and the primary owner of a project is responsible for its operation, fault response, and gray‑release practices.

The department operates without dedicated product managers; engineers identify problems themselves, and consensus drives the work direction. New technologies are adopted only after proven success at comparable scale and after thorough understanding of the codebase, which explains the preference for Python, C, and Go over Java.