Weibo’s Third-Generation Technical Architecture Overview

Preface

In March 2014, Sina Weibo reported 143 million monthly active users; the first minute of 2014 saw 808,298 posts, requiring a highly available, high‑concurrency, low‑latency backend system.

The first‑generation architecture was a LAMP stack using MyISAM, PHP, and Memcache.

As the platform grew, the second generation modularized and service‑oriented the system, replacing PHP with Java and forming an SOA architecture that supported business development for a long period.

After extensive refactoring, online operation, and consolidation, a third‑generation architecture emerged.

Below is a simplified core‑business diagram; the third‑generation technical system is designed to ensure rapid, efficient, and reliable delivery of new products and features on the core Weibo business.

Third-Generation Technical System

The third‑generation system uses orthogonal decomposition: horizontally it adopts a typical three‑tier model—interface layer, service layer, and resource layer; vertically it is further divided into business architecture, technical architecture, monitoring platform, and service‑governance platform.

As shown, the orthogonal decomposition splits the diagram into 3 × 4 = 12 zones, each representing the intersection of a horizontal and vertical dimension; for example, zone 5 mainly implements the technical architecture of the service layer.

The following sections detail the design principles of the horizontal and vertical dimensions, with particular focus on the components in zones 4, 5, and 6.

Horizontal Layering

Horizontal division is a fundamental practice in the design of large‑scale internet backend systems and appears in every generation of the platform.

The interface layer implements interactions with web pages and mobile clients, defines a unified API contract, and provides three core services: Feed, User‑Relationship, and Messaging (private, group, and chat).

The service layer modularizes core business logic. It is split into two categories: atomic services that do not depend on any other service (e.g., short‑link service, ID generator), and composite services that combine atomic services and business logic (e.g., Feed service, Messaging service) and thus depend on other services.

The resource layer stores data models, including common cache resources (Redis, Memcached) and persistent stores (MySQL, HBase), as well as distributed file systems (TFS) and Sina S3.

Horizontal layering has a clear top‑down dependency: upper layers depend on lower layers, while lower layers do not depend on upper layers.

Correspondingly, the servers in the Weibo system include three types: front‑end machines (providing API services), queue machines (handling upstream business logic and data writes), and storage nodes (memcache, MySQL, MCQ, Redis, HBase, etc.).

Vertical Extension Technical Architecture

Through continuous evolution, the platform has built many middleware products driven by business needs, forming a complete technical framework that greatly improves development efficiency and operational stability.

Unlike the horizontal top‑down dependencies, the vertical direction uses the technical framework as a foundational support point that influences the business architecture, monitoring platform, and service‑governance platform.

Interface Layer Web V4 Framework

The interface framework simplifies and standardizes API development, built on Spring AOP and a customized Jersey implementation. It uses annotations to define URLs and parameters, and includes built‑in authentication, rate limiting, access logging, degradation, and automatic Bean‑JSON/XML serialization.

Service Layer Framework

The service layer mainly involves an RPC remote‑call framework and a message‑queue framework, which are the two most widely used frameworks on the Weibo platform.

MCQ Message Queue

The message queue provides a FIFO communication mechanism; internally it is commonly used to asynchronously write data to persistence, allowing batch DB writes and improving front‑end response time. It also feeds real‑time data to search, big‑data, and business‑operation teams.

Weibo’s MCQ (SimpleQueue Service Over Memcache) is based on the Memcache protocol, persists messages to BerkeleyDB, supports only get/set commands, is easy to monitor (stats queue), offers rich client libraries, and has performance many times higher than generic MQs.

Motan RPC Framework

Weibo’s Motan RPC service uses Netty as the underlying network engine. It supports Hessian and Java serialization, and protocols such as Motan, HTTP, TCP, and MC. Motan is heavily used internally and provides mature solutions for robustness and service governance. High availability and load‑balancing are achieved via Config, supporting flexible FailOver and FailFast HA strategies as well as Round Robin, LRU, Consistent Hash, etc. Service governance includes full call‑chain data, performance metrics, response time, QPS, and standardized error/exception logs.

Resource Layer Framework

The resource layer contains many frameworks, including MySQL/HBase DAL middleware, custom counters, and distributed cache proxies. Two highlighted components are the Object Store and SSDCache.

Object Store

The Object Store supports convenient serialization and deserialization of Weibo objects: during serialization, JVM objects are written to HBase and assigned a unique ObjectID; retrieval is performed via this ObjectID. It supports arbitrary object types and protocols such as PB, JSON, and binary. The largest use case is unifying videos, images, and articles as objects, defining dozens of object types and a standard metadata schema. Object content is stored in the Sina S3 object storage, with metadata containing the download URL.

SSDCache

With the widespread adoption of SSDs, their superior I/O performance is used to replace traditional SATA/SAS disks. Common scenarios include: (1) replacing MySQL disks (even without a dedicated SSD‑optimized MySQL version, SSDs can bring ~8× IOPS improvement); (2) replacing Redis disks to boost performance; (3) accelerating static resource loading in CDNs.

Weibo extends the traditional Redis/MC + MySQL stack to Redis/MC + SSDCache + MySQL, using SSDCache as an L2 cache. This reduces the high cost and limited capacity of MC/Redis, and alleviates database pressure caused by cache penetration.

Vertical Monitoring and Service Governance

As service scale and complexity increase, tracing dependencies becomes difficult, and operations become harder. Inspired by Google’s Dapper and Twitter’s Zipkin, the platform built its own large‑scale distributed tracing system, WatchMan.

WatchMan Distributed Tracing System

Like other large internet applications, Weibo consists of many distributed components. Each HTTP request from a browser or mobile client traverses multiple business systems or components, leaving a footprint. Collecting and analyzing these footprints is crucial for troubleshooting and performance optimization. The system also aggregates performance data to enable flow control, degradation, and SLA‑based feedback.

WatchMan is designed to be low‑intrusive: logging points are placed only in middleware layers (interface framework, RPC framework, other resource middleware), keeping the instrumentation transparent to business code.

The system is used by both operations and business teams to achieve distributed service governance, including service scaling, degradation, traffic switching, and gray releases.

Conclusion

Today the technical framework plays an increasingly important role in driving platform upgrades, business development, and operational services. This article is limited in length and does not cover all details; future articles will continue to introduce the design principles and architecture of core middleware.

About the Author

Wei Xiangjun (Weibo: @WeiXiangjun_Weibo), a graduate of Beijing University of Posts and Telecommunications, currently a Weibo platform architect. He previously worked at Microsoft, Kingsoft Cloud, and Sina Weibo, focusing on system architecture design, audio‑video communication systems, distributed file systems, and data mining.

Thanks to Zang Xiutao for proofreading.

Source: InfoQ