Unveiling Modern Big Data Architecture: Key Technologies and Trends

The lecture introduces the technical architecture of big data, covering four main topics: traditional databases and data warehouses, Hadoop and its ecosystem, commercial big‑data technology architecture, and big‑data commercial products.

Traditional databases and data warehouses

Hadoop and its ecosystem

Commercial big‑data technology architecture

Big‑data commercial products

Understanding the evolution of database systems—from the 1960s through three generations—reveals their foundational role in information infrastructure, the rise of relational algebra, SQL, and the massive software industry built around DBMS.

With exploding data volume, variety, and velocity, traditional data warehouses face challenges: rapid data growth, increasing data source types (including unstructured data), the need for database virtualization to unify hundreds of heterogeneous databases, and the demand for built‑in search and data‑mining capabilities.

Hadoop and Spark have become the core of next‑generation data‑warehouse solutions, addressing these challenges through distributed processing.

Big‑data management systems can be classified into four categories:

MPP parallel databases and in‑memory databases

Hadoop‑based open‑source big‑data systems

Hybrid clusters of MPP databases and Hadoop

Hybrid in‑memory computing with Hadoop

Key conclusions include the dominance of Hadoop/Spark for distributed processing, convergence of structured and unstructured data platforms, the gradual replacement of MapReduce by Spark, continued relevance of SQL (enhanced by SQL‑on‑Hadoop/Spark), and the rise of SQL‑centric big‑data systems challenging traditional databases.

Big Data Computing

Computing models such as MapReduce, Spark’s RDD, and graph‑parallel abstractions address diverse big‑data workloads, but no single model fits all scenarios; thus, multiple high‑level computation models have emerged.

Big Data Analysis

Big‑data analysis aims to extract maximum insight from data through statistical analysis, data mining, and machine learning.

Statistical analysis applies descriptive and inferential statistics, including regression, factor analysis, clustering, and discriminant analysis.

Data mining discovers patterns using algorithms like C4.5, k‑means, SVM, Apriori, EM, PageRank, AdaBoost, kNN, Naïve Bayes, and CART, as well as neural networks and genetic algorithms.

Machine learning designs algorithms that automatically learn from data, enabling high‑performance analytics on large clusters and supporting iterative, fault‑tolerant processing distinct from traditional OLAP.

Statistical methods influence data mining, while machine learning and databases underpin data mining techniques.

Data Visualization

Visualization translates abstract data into graphical forms, essential for interpreting massive datasets; traditional tools like spreadsheets cannot handle big‑data scale, prompting research into scalable visual analytics.

Big Data Technology Providers

Major vendors offering end‑to‑end big‑data solutions include IBM, Microsoft, Google, Amazon, Baidu, Tencent, Alibaba, Huawei, Inspur, and ZTE. Emerging startups focus on innovative Hadoop‑based platforms, often positioned as Visionaries in Gartner’s Magic Quadrant.

Leading Hadoop distributors such as Cloudera, Hortonworks, MapR, Informatica, Microsoft, and Oracle dominate the market, while companies like Vertica, Greenplum, IBM Big Insights, Yonyou, and StarRocks contribute specialized capabilities.

Traditional leaders such as Oracle and Teradata remain influential, but the shift toward Hadoop‑centric architectures accelerates, with domestic firms like StarRocks aiming to become future leaders.