Baidu's Third‑Party Sentiment Feedback System: Architecture, Data Capture, Cleaning, and Output

Case Background Software development has evolved from waterfall to agile and then lean, each paradigm shifting quality priorities. In the 80s‑90s, abundant hardware and rising software demand made waterfall dominant, emphasizing comprehensive test coverage at low cost. The rise of the internet prompted a shift to agile, with continuous integration and automation, followed by lean practices that close the feedback loop between product and user.

The presentation focuses on how Baidu built a feedback system and quality closed‑loop, especially the architecture for third‑party public‑opinion data collection, cleaning, and delivery.

Solution Overview The engineering architecture consists of three layers: data capture , data cleaning , and data output . Capture must handle format changes and anti‑scraping measures; cleaning balances latency, recall, and cost; output bridges the infrastructure with business needs.

A. Data Capture Baidu aggregates public‑opinion sources such as Weibo, Baidu Tieba, app stores, news, search, and forums. A scheduler distributes URLs and configuration to independent crawlers, handling retries, IP/ bandwidth allocation, and adaptive response to throttling or blocking. Human‑in‑the‑loop interventions adjust configurations when sites change structure or impose bans, keeping maintenance costs low.

B. Data Cleaning Cleaning proceeds in three stages: filtering, relevance determination, and sentiment tagging. Filtering removes marketing boilerplate and spam using sampling‑based clustering and O(N) string matching. Relevance analysis disambiguates homonyms (e.g., “糯米团”) via context‑aware classification. Sentiment analysis includes sarcasm and contextual detection, feeding back into model training through a small manually‑checked sample (1‑5%).

C. Data Output Cleaned data are stored in a multi‑key‑multi‑value feedback data warehouse, indexed by time, product line, source, relevance, and sentiment. Real‑time incremental indexes expose a standard API for downstream consumers to query by various dimensions, enabling use cases such as monitoring, competitor analysis, and rapid issue recall.

Application Scenarios The system supports three main scenarios: (1) public‑opinion monitoring (e.g., detecting search traffic hijacking or campaign impact), (2) competitor analysis (e.g., comparing refund‑process sentiment across platforms), and (3) quality‑issue closure (e.g., spotting inappropriate search results before they spread). These enable product, risk, and brand teams to act within hours.

Conclusion Testing teams should consider integrating third‑party sentiment data to close the quality loop, complementing traditional testing and monitoring. The presented architecture demonstrates a scalable, maintainable approach to collect, clean, and serve feedback at Baidu’s scale.