This article explains how logs—simple, append‑only, time‑ordered records—serve as the core abstraction behind databases, distributed systems, data integration pipelines, and modern stream‑processing platforms such as Kafka and Hadoop, illustrating their design, scalability, and practical challenges.
This article explains how logs—simple, append‑only, time‑ordered records—serve as the fundamental abstraction behind databases, distributed systems, data integration pipelines, and stream‑processing platforms like Kafka and Hadoop, illustrating their role in ordering, replication, scalability, and real‑time analytics.
This article chronicles Baidu’s eight‑year journey from early Hadoop adoption to advanced MPI, DAG engines, and real‑time streaming platforms, detailing architectural milestones, performance optimizations, and practical lessons for large‑scale offline and online data processing.
The article examines various distributed storage design approaches—decentralized (GlusterFS), centralized (Hadoop), database‑based (GridFS and HBase), and metadata‑bypassing (FastDFS)—detailing their advantages, drawbacks, and practical considerations for cloud storage systems.
This article provides a comprehensive overview of major big‑data platforms such as Hadoop, Spark, Flink, Kafka, and related ecosystem components, explaining their core concepts, storage models, processing frameworks, and architectural patterns for handling massive, distributed datasets.
This article provides a concise, practical walkthrough for installing and configuring Apache Hive on a Hadoop cluster, covering prerequisite HDFS and MapReduce setup, downloading Hive, extracting files, setting environment variables, configuring XML files, starting Hive, and verifying the installation with simple commands.
This article chronicles Suning’s journey from early SAP‑based data warehouses to a modern, open‑source big data platform featuring real‑time collection, Hadoop‑Hive offline processing, Storm‑based streaming, and a visual development environment, highlighting how each layer addresses growing data volume, variety, and business demands.
The article examines the evolution of database technologies—from traditional relational databases and their ACID guarantees to NoSQL, NewSQL, and Hadoop—illustrating how a gaming company can combine these solutions to handle massive online traffic, ensure data integrity, and enable advanced analytics.
This article explains Hadoop’s tiered storage concept, describing how data is classified by temperature—hot, warm, cold, frozen—and automatically moved across disk and archive layers to optimize cost and performance, with examples from Hadoop versions and eBay’s large‑scale deployment.
This article explains why many companies over‑invest in Hadoop, outlines how to evaluate data size, growth, and relevance, and presents practical alternatives such as archiving, data sampling, database sharding, and hiring business‑savvy analysts before committing to a Hadoop deployment.
This article provides a comprehensive overview of the Hadoop family—detailing more than twenty open‑source projects, their core functions, and a structured learning roadmap to help developers master Hadoop, Hive, Pig, HBase, Zookeeper, Mahout, and related tools.
This article outlines five essential steps to avoid Java heap space errors in Hadoop MapReduce by estimating memory consumption, verifying JVM availability and settings, limiting swap usage, and configuring instance slot numbers below the JobTracker's calculated values, ensuring stable cluster performance.
The article explains Apache Spark’s memory‑based distributed computing model, its advantages over Hadoop’s MapReduce, key features, fault tolerance, deployment modes, ecosystem components, and the scenarios where Spark is most effective for large‑scale data analytics.
This article explains Hadoop 2.0's High Availability architecture, comparing the NFS and Quorum Journal Manager methods, detailing their principles, failover mechanisms, and practical tips for reliable NameNode redundancy in big‑data deployments.
This article walks through the essential steps to install and configure Hadoop 2.0 on a two‑node Linux cluster, covering version selection, directory setup, core XML files, YARN settings, service startup, verification commands, and basic troubleshooting tips.
The interview with Fourinone creator Peng Yuan reveals the framework's evolution from a parallel computing library to a lightweight 220KB distributed system with its own NoSQL database engine, compares it to Hadoop, discusses the CoolHash design, and outlines Huawei's FusionInsight big‑data platform, while providing open‑source repository links.
The interview with Fourinone founder Peng Yuan explores the framework's evolution from a parallel computing project to a 220 KB distributed system with its own NoSQL database engine CoolHash, compares it to Hadoop, and discusses its open‑source release, technical design choices, and real‑world deployments in finance and enterprise environments.
This article provides a concise overview of several widely-used distributed file systems—including Lustre, Hadoop, MogileFS, FreeNAS, FastDFS, NFS, OpenAFS, MooseFS, pNFS, and GoogleFS—highlighting their origins, key features, supported platforms, and typical use cases.
This article explains how to evaluate Hadoop workloads and select balanced hardware—covering storage, compute, network, and memory specifications for master and worker nodes—to build an efficient, cost‑effective Hadoop cluster.
The article warns that mastering Free Open Source Software—from cloud and big‑data tools like OpenStack, Hadoop and NoSQL to web technologies such as Drupal, PHP, HTML5 and jQuery, mobile app development, popular programming languages, and security certifications—will become essential for IT job seekers, prompting universities to embed FOSS curricula.
