Scaling Sina's News Comment System: From 3.0 to 5.0 – Key Backend Lessons

Origin of the News Comment System

Sina introduced comment functionality early on, initially using a simple Perl script. The earliest commentable news article dates to April 7, 2000. Although the original comment URLs are now dead, the data remains stored in databases, and commenting has become a standard feature on all Chinese news portals.

Comment System 3.0

Around 2003 the system was upgraded to version 3.0, running on a single Dell 6300 server with Solaris, MySQL, Apache, and CGI programs written in C++. The caching module stored pages in files, each covering two pages of comments with one overlapping page to simplify updates. New comments triggered a single file‑append operation, while deletions required clearing subsequent cache files. This design worked until sudden traffic spikes overloaded the single server, exposing severe database contention on a massive MyISAM table.

To alleviate the bottleneck a low‑spec FreeBSD server was added for database off‑loading; old comments were moved weekly to keep the primary table size manageable. This temporary fix extended the life of 3.0 for a few months.

Comment System 4.0 Launch

By 2004 the 3.0 architecture could no longer handle rising traffic, prompting the 4.0 project with the core requirement “no downtime.” A dual‑system migration kept the existing database unchanged while gradually shifting traffic to the new platform.

Phase 1: File System Replaces Database, ICE Distributed System

Instead of direct DB writes, each news article’s comments were stored in an indexed file and a data file, allowing concurrent file‑level reads/writes and reducing load on MySQL, which was relegated to archival tasks. A simple file‑based message queue was built for asynchronous communication, and ICE was chosen as the RPC framework for inter‑module calls.

Architecture diagrams (Fig 3‑5) show a C++‑based system with MySQL 4.0, ICE, and a new file system layer. The cluster grew from a single machine to a five‑node setup, but intensive random I/O on Ext3 eventually crippled performance.

Attempts to switch to ReiserFS mitigated some I/O issues, yet the underlying design still suffered from massive file‑system load.

Phase 2: Full System Asynchrony and Index Paging Optimization

The front‑end moved from Apache + CGI to static HTML pages that loaded comment data via AJAX (XMLHTTP). This decoupled user actions from server processing, allowing cache updates to be handled asynchronously through a message queue.

Pagination was initially handled by loading entire index files into memory via mmap, sorting, and filtering. This worked for thousands of comments but became prohibitive at the tens‑of‑millions level, leading to a hybrid precise‑fuzzy paging strategy that avoided full‑table sorts.

Memcached was later introduced to store comment bodies, leaving only index data on the file system, dramatically reducing I/O pressure.

High‑Traffic UGC System Design Summary

Key design principles emerged: use queues to smooth write spikes, layered caching (file system, database, memory) to bring data close to the consumer, and sharding/partitioning to keep active data sets small. Separate hot (recent) from cold (historical) data, split indexes from entity data, and isolate core business tables from auxiliary features.

MySQL replication remains a common scaling method, but its single‑threaded write limitation is still a bottleneck. Caching should be applied only when a read‑many/write‑few pattern exists, and cache updates should favor overwrite semantics to avoid error accumulation.

Comment System 5.0 Plan

Post‑2010 the 5.0 design addressed 4.0’s shortcomings: comment IDs switched from auto‑increment integers to UUIDs, eliminating write‑after‑commit delays; the database schema was re‑engineered for high‑frequency operations with clear hot/cold splits; an in‑memory NoSQL cache replaced file‑based page caches; the platform migrated to Sina’s internal cloud; and the codebase was rewritten in Python, removing long compile cycles.

News Comment Product Summary

News comments were the premier UGC channel in China’s early 2000s, reflecting public opinion on major events. Despite strict moderation and missed opportunities for real‑name integration, the system remains a pivotal case study in large‑scale, user‑generated content engineering.