Why Big Data Is Driving the Next Technological Revolution

01 This Era Belongs to Big Data

In the early days of speech recognition, machine translation, image recognition and natural language understanding, researchers split into two camps: one clinging to traditional AI that mimics humans, the other advocating data‑driven methods. Data‑driven approaches quickly dominated speech and language tasks, while image and translation lagged because of scarce training data.

Before the Internet, image datasets never reached millions and bilingual corpora were limited to the Bible and a few UN documents. After the Internet’s rise in the 1990s, data became abundant; from 1994 to 2004, speech error rates halved and machine‑translation accuracy doubled, with roughly 80% of the gain coming from more data.

Data‑driven methods began in the 1970s, grew steadily in the 80s‑90s, and exploded in the 21st century as the web supplied massive, multi‑dimensional data, turning quantity into quality and enabling computers to solve many tasks once thought to require human intelligence.

As data from different domains intersected, their relationships strengthened, giving rise to the concept of big data.

02 Smartphones and the Internet Keep Changing the World

Real‑time data is not essential, but it enables capabilities impossible before, such as intelligent traffic management. Before smartphones and connected cars, city traffic centers suffered 20‑minute delays. Widespread smartphones that share location data let map services like Google or Baidu obtain live population flows, distinguishing pedestrians from vehicles and providing near‑instant traffic updates.

Because the same company collects and serves the data, latency is minimal, allowing more accurate predictions of future congestion and better route planning.

Big data represents a shift in thinking: massive data changes the way we approach problems, turning uncertain, intelligent tasks into data problems that can be solved with information theory.

03 Big Data Is a Revolution in Thinking

Historically, mechanical thinking—simple models built on deterministic laws—guided science from Ptolemy to Newton. It emphasized predictability and causality. However, in the information age, many phenomena cannot be captured by simple causal rules; uncertainty dominates.

Data‑driven thinking accepts uncertainty and uses large, multi‑dimensional datasets to uncover correlations (mutual information) and validate hypotheses (cross‑validation), replacing the need for strict causal models.

Information theory shows that with complete data, training and testing sets converge, eliminating the “small‑sample” problem and making data‑driven methods universally applicable.

04 The Essence of Big Data

Big data’s core is information that reduces uncertainty. When data volume is insufficient, progress stalls; the first field to accumulate enough data (speech recognition) advanced fastest. Multi‑dimensional data provides mutual information and cross‑validation, while data completeness ensures models see virtually every possible event.

Thus, big data’s scientific foundation is information theory: using information to eliminate uncertainty.

05 Internet Companies’ Data Competition

Search engines now rely heavily on click‑through data; the “click model” accounts for 70‑80% of ranking decisions. Larger data volumes improve long‑tail query performance, turning the industry from a technology race into a data race.

Companies acquire data through browsers, toolbars, and input methods, allowing even small search players to boost accuracy if they capture enough user interactions.

Data‑driven thinking also reshapes business models: Amazon leverages comprehensive customer data to personalize offers, while traditional retailers lack such insight.

06 Existing Industries + Moore’s Law = New Industries

Just as past industrial revolutions combined existing sectors with new technologies, the current era combines big data (and machine intelligence) with established industries to create new business models. The formula is simple: existing industry + big data = new industry, or existing industry + machine intelligence = new industry.

Historical analysis shows that technological revolutions drive commercial model changes, push manufacturing toward service‑oriented, data‑rich enterprises, and make data a decisive competitive advantage.