This article describes how Yahoo integrated deep learning into its massive Hadoop ecosystem by adding GPU nodes, using YARN and Spark to run Caffe at scale, and presents performance results on AlexNet and GoogLeNet alongside open‑source contributions.
Spark’s big‑data analytics ecosystem comprises core components such as the in‑memory RDD data structure, Streaming for real‑time processing, GraphX for graph analytics, MLlib for machine‑learning, Spark SQL for querying, the Tachyon file system, and SparkR, each enabling scalable, distributed computation.
The 2015 Suning Big Data Meetup in Nanjing gathered industry experts and researchers to showcase Suning's data platform architecture, Intel's Spark advancements, ZTE's DAP system, and a unified Octopus programming model, emphasizing open knowledge sharing and practical big‑data solutions.
This article provides a comprehensive overview of major big‑data platforms such as Hadoop, Spark, Flink, Kafka, and related ecosystem components, explaining their core concepts, storage models, processing frameworks, and architectural patterns for handling massive, distributed datasets.
This article explains how Apache Spark improves on Hadoop MapReduce by keeping intermediate data in memory, introduces the core RDD abstraction, compares Spark’s transformations and actions with Hadoop, and shows how Spark can run on Standalone, YARN, and various programming languages such as Scala, Java, and Python.
This article reviews Spark 1.2’s major enhancements—including the External Data Source API, column pruning, predicate pushdown, and in‑memory columnar storage—while also detailing Baidu’s large‑scale Spark deployments, its custom high‑performance Shuffle service, and the integration of Spark with the Tachyon memory file system.
The article explains Apache Spark’s memory‑based distributed computing model, its advantages over Hadoop’s MapReduce, key features, fault tolerance, deployment modes, ecosystem components, and the scenarios where Spark is most effective for large‑scale data analytics.
