Designing a Real‑Time Big Data Sentiment System on Alibaba Cloud: From Lambda to Lambda‑Plus

With the rapid growth of the internet, massive amounts of media, e‑commerce orders, and user comments generate valuable sentiment data that must be processed in real time.

Key requirements for a big‑data sentiment system include:

Real‑time ingestion of massive raw data : a crawler collects web pages, de‑duplicates them before and after fetching, and extracts sub‑pages.

Processing of raw web data : transform unstructured pages into structured records such as titles, abstracts, and product reviews.

Structured sentiment analysis : classify and label content with sentiment tags, detect hot topics, influence, propagation paths, user profiling, and generate alerts for critical events.

Storage and interactive query : support full‑text search and flexible multi‑field queries for both analysis and business decision making.

Real‑time alerting : trigger notifications when major sentiment events occur.

System Design Overview

The architecture first presents a typical Lambda architecture, which combines batch processing (e.g., Hadoop, Spark) with stream processing (e.g., Spark Streaming, Flink). Data flows from a Kafka queue to both a batch layer stored in HDFS/HBase and a real‑time layer that updates a serving store.

The Lambda architecture offers unlimited historical replay but requires maintaining separate storage and processing pipelines.

To simplify, the Kappa architecture keeps only a long‑term log in Kafka, reprocessing data from the log when needed, thus eliminating the batch store at the cost of limited retention.

Lambda‑Plus Architecture

Combining the strengths of Lambda and Kappa, Lambda‑Plus uses a single distributed database that supports both random access and sequential log consumption. This enables one codebase to serve both batch and streaming workloads while storing all data in one place.

Cloud Implementation on Alibaba

Alibaba Cloud provides two managed services that realize the Lambda‑Plus design:

Tablestore : a multi‑model distributed database used for all storage layers (raw pages, structured data, metadata, and sentiment results). Its channel service exposes database logs as a queue, enabling real‑time consumption.

Blink : a unified stream‑batch engine that reads from Tablestore logs, performs real‑time extraction, tokenization, OCR, sentiment tagging, and writes results back to Tablestore or Elasticsearch for flexible querying.

Advantages of this stack include:

Deep integration between Tablestore and Blink eliminates the need for separate source, dimension, and sink tables.

Only two managed products replace six open‑source components, drastically reducing operational overhead.

Developers focus on business logic; data movement is handled by the platform.

The database serves both as a storage engine and a queue, simplifying architecture.

One codebase supports both real‑time and batch processing, enabling efficient sentiment feedback loops.

Challenges of traditional open‑source stacks are also highlighted, such as the complexity of maintaining Kafka, HBase, Spark, Flink, and Elasticsearch together, data consistency issues, and the need for dual APIs.

References:

Lambda architecture – https://mapr.com/tech-briefs/stream-processing-mapr/

Kappa architecture – https://www.oreilly.com/ideas/questioning-the-lambda-architecture

Lambda vs. Kappa – https://www.ericsson.com/en/blog/2015/11/data-processing-architectures--lambda-and-kappa