Comprehensive Overview of Big Data Open‑Source Frameworks

System Platforms

Hadoop : The foundational distributed storage and compute framework originating from Nutch and Google’s GFS/MapReduce papers; includes HDFS, MapReduce, and YARN. Stable 2.x releases are common, with 3.2.0 as the latest.

Cloudera CDH : A packaged distribution of Hadoop and related projects with a Web UI (Cloudera Manager) for deployment and monitoring; widely adopted in enterprises for its ease of installation and management.

Hortonworks HDP : An open‑source Hadoop distribution that adds native Windows support via Ambari; known for flexibility but less stability than CDH.

Cluster Management & Monitoring

Cloudera Manager : Web‑based tool for automated cluster installation, centralized management, and monitoring (non‑open‑source).

Hue : A Python/Django‑based UI for interacting with Hadoop components (HDFS, Hive, HBase, etc.).

Apache Ambari : Provides installation, configuration, and Web UI for Hadoop ecosystem management.

Dr.Elephant : LinkedIn’s performance‑monitoring tool for Hadoop and Spark jobs, automatically collects metrics for tuning.

Ganglia and Zabbix : General‑purpose monitoring systems for large clusters.

Apache Eagle : Real‑time security and performance monitoring with stream‑processing and machine‑learning based profiling.

File Systems

HDFS : Hadoop’s distributed file system; high throughput but poor for low‑latency or many small files.

GPFS , Ceph , GlusterFS , Swift , BeeGFS , Alluxio : Various distributed or parallel file systems offering POSIX semantics, high availability, or memory‑centric access.

Resource Scheduling

YARN : Hadoop 2.x’s resource manager, separating resource allocation (ResourceManager) from job execution (ApplicationMaster, NodeManager).

Apache Mesos : General‑purpose cluster manager; originally from UC Berkeley, later superseded by Kubernetes in many deployments.

Coordination Frameworks

ZooKeeper , Etcd , Consul : Distributed consensus services used for configuration, leader election, and service discovery; ZooKeeper uses ZAB, Etcd uses Raft, Consul adds health checks and multi‑datacenter support.

Data Storage

HBase : Column‑oriented NoSQL store built on HDFS; Google Bigtable implementation.

Cassandra and ScyllaDB : Wide‑column stores with peer‑to‑peer replication; ScyllaDB claims 10× higher throughput.

MongoDB : Document‑oriented database with rich query language and indexing.

Redis : In‑memory key‑value store supporting persistence, pub/sub, and data structures.

Ignite , Apache Arrow , Geode , Kudu , CarbonData : Various in‑memory, columnar, or hybrid storage solutions offering low‑latency analytics.

Data Processing

MapReduce : Classic batch processing; limited by disk I/O.

Apache Spark : In‑memory engine supporting batch, streaming, SQL, ML, and graph processing.

Apache Flink : Stream‑first engine with true batch‑stream unified processing and exactly‑once semantics.

Apache Storm : Real‑time stream processing; largely superseded by Spark/Flink.

Apache Tez , Samza , Apex , Beam , Heron : Specialized engines for DAG execution, stream processing, or unified pipelines.

Query & Analysis

Hive and SparkSQL : SQL‑on‑Hadoop layers; Hive translates queries to MapReduce (or Tez/Spark) jobs.

Presto : Low‑latency distributed SQL engine for interactive analytics.

Kylin : Cube‑based OLAP engine delivering sub‑second query speeds.

Impala : MPP SQL engine inspired by Google Dremel.

Druid and ElasticSearch : Real‑time analytics and full‑text search engines.

HAWQ , Lucene , Solr , Phoenix : Additional query or indexing solutions.

Data Collection

Flume , Filebeat , Logstash , Chukwa : Log and event ingestion pipelines.

Data Exchange

Sqoop : Bulk transfer between relational databases and Hadoop.

Kettle (Pentaho), DataX , NiFi : ETL and flow‑based data movement tools.

Messaging Systems

Apache Pulsar : Distributed pub/sub and queue system with BookKeeper storage.

Kafka : High‑throughput log‑based messaging platform.

RocketMQ , ActiveMQ , RabbitMQ : Additional brokers with varying protocols and durability guarantees.

Task Scheduling

Azkaban , Oozie , Airflow : Workflow orchestration tools ranging from simple batch (Azkaban) to DAG‑based Python pipelines (Airflow).

Data Governance

Apache Ranger : Centralized security policy engine for Hadoop components.

Sentry : Fine‑grained access control for Hive, Impala, and HBase.

Apache Atlas : Metadata catalog and lineage tracking, often integrated with Ranger.

Visualization

Kibana : Dashboard for Elasticsearch data.

D3.js and ECharts : JavaScript libraries for custom visualizations.

Data Mining & Machine Learning

Mahout and Apache MADlib : Scalable ML libraries for clustering, classification, and regression.

Spark MLlib : In‑memory ML library built on Spark.

TensorFlow and Keras : Deep‑learning frameworks for neural networks.

Cloud Platforms

AWS S3 , Google Cloud Platform (GCP) , and Microsoft Azure provide object storage and compute services that can complement or replace on‑premise Hadoop clusters.