Unpacking the Core Technologies Behind Modern Big Data Platforms

Big Data Collection

Data collection gathers data from various sources and stores it in the data storage layer, sometimes performing simple cleaning. Common sources include website logs (collected by Flume agents to HDFS), business databases (MySQL, Oracle, SQL Server) synchronized via DataX or Flume, FTP/HTTP sources, and manually entered data via simple interfaces.

Big Data Storage and Analysis

HDFS is the primary storage solution for data warehouses. Offline analysis can be performed with Hive, which offers rich data types, built‑in functions, ORC compression, and SQL support, providing far more efficient processing than MapReduce. Hadoop’s MapReduce is also available for Java‑centric development. Spark, increasingly popular, delivers superior performance and integrates well with Hive and YARN, allowing SparkSQL for analysis without a separate Spark cluster.

Big Data Sharing

After analysis, results are stored in relational or NoSQL databases for sharing. DataX can synchronize results from HDFS to these targets, and real‑time computation modules may write directly to the sharing layer.

Real‑time Data Computation

To meet real‑time requirements, a distributed, high‑throughput, low‑latency framework is needed. The article chooses Spark Streaming over Storm for simplicity, using Flume to collect logs, stream them to Spark Streaming, and store aggregated results in Redis for immediate access.

Task Scheduling and Monitoring

Data platforms involve many tasks (collection, synchronization, analysis) with complex dependencies. A robust scheduling and monitoring system is essential to orchestrate and track these tasks.