Big Data and Prediction: Insights from Baidu Research Lab

On August 16, Shen Zhiyong, a scientist from Baidu Research Institute’s Big Data Laboratory, delivered a technical talk at the 53rd Baidu Technology Salon. He described the goal of building an online intelligent system that integrates monitoring, anomaly detection, diagnosis, and prediction to support operations and maintenance.

Big data refers to data volumes so massive that conventional software tools cannot capture, manage, process, and transform them into actionable business insights within a reasonable time. It is considered a frontier of innovation and productivity, spawning numerous prediction technologies.

Big Data and Prediction

Shen explained that prediction serves as a basis for decision‑making and planning, citing everyday examples such as weather forecasts for leisure activities and quantitative forecasts like stock prices that provide informational advantage.

Prediction at Baidu can be classified into qualitative (e.g., weather) and quantitative (e.g., stock prices) categories.

The laboratory’s core method for prediction is time‑series analysis, which underpins Baidu’s tourism forecasting service.

He shared an anecdote: early versions of Baidu’s voice assistant could not answer questions like “How many people will be at the Forbidden City tomorrow?” This limitation highlighted the need to look beyond the present and led to the development of tourism prediction.

Besides time‑series analysis, causal analysis (stronger than correlation) and simulation analysis (directly modeling future scenarios) are also frequently employed in big‑data prediction.

Big‑Data Era: Opportunities and Challenges

Baidu is one of the earliest companies in China to conduct big‑data research. Its Big Data Lab (BDL), led by world‑renowned machine‑learning scholar Prof. Zhang Tong, focuses on the Baidu Brain and related big‑data technologies. Shen emphasized that big data is as essential to Baidu as air; without it, the company could not operate.

While the abundance of data provides rich sources for machine‑learning algorithms, it also introduces challenges in data processing and handling heterogeneous data types.

Case Studies: Scenic‑Spot Forecasting and Gaokao Essay Prediction

Shen presented two Baidu product cases that rely on big‑data prediction. The scenic‑spot forecast uses time‑series analysis, incorporating historical visitor counts, seasonal patterns, weather conditions, and search query volume to model and predict daily crowd levels.

The Gaokao essay prediction is more complex. By analyzing a large corpus of high‑quality essays, the team builds a topic model and combines it with current trends to infer the likely direction of upcoming exam prompts.

In conclusion, Shen stated that prediction is merely the entry point for Baidu’s big‑data lab; the ultimate aim is to develop an online intelligent system that simulates human analysis and decision‑making.

The session attracted over 300 attendees, who engaged in an active Q&A session with Shen, creating a vibrant learning atmosphere.