This article explains how to efficiently move large volumes of data from Hive to HBase by leveraging HBase's bulkload mechanism, detailing the original MapReduce workflow, its performance bottlenecks, and a rewritten Spark‑based solution that simplifies ETL, improves partitioning, and achieves several‑fold speedup.
This article explains why MySQL queries become slow, explores index-related pitfalls and optimization techniques, and then compares ElasticSearch and HBase as complementary solutions for large‑scale data and search scenarios, offering practical tips and code examples.
This article walks through a complete real‑time recommendation system built on Apache Flink, detailing its v2.0 architecture, modules for user behavior, interest, and product profiling, the recommendation algorithms (hot‑list, collaborative filtering, item similarity), and step‑by‑step Docker deployment of MySQL, Redis, HBase, and Kafka.
This article walks through a complete Flink‑based recommendation system, detailing its v2.0 architecture, recommendation algorithms, front‑end and back‑end components, and step‑by‑step Docker deployment of MySQL, Redis, HBase, and Kafka services.
This article explains common causes of slow MySQL queries, how proper indexing and lock handling can improve performance, introduces Elasticsearch’s inverted‑index advantages and suitable use cases, and outlines HBase’s column‑family storage model and row‑key design for large‑scale data.
This article explores the common causes of MySQL slow queries, explains index usage and pitfalls, introduces ElasticSearch architecture and its fast search capabilities, and outlines HBase's column‑family storage model and suitable use cases for large‑scale data systems.
This article explains why MySQL queries become slow, how indexes work and fail, the impact of MDL locks, large‑table challenges, sharding and read‑write splitting strategies, then introduces Elasticsearch’s search capabilities and HBase’s column‑family storage, offering practical guidance for each technology.
The article examines the strengths and weaknesses of combining HBase and Elasticsearch for massive data storage and retrieval, outlines three integration patterns and their challenges, and presents Alibaba Cloud's Lindorm Searchindex as a SQL‑driven, low‑cost, strongly consistent solution that simplifies development and improves performance.
This article presents a comprehensive walkthrough of a Flink‑powered recommendation system, detailing its v2.0 architecture, module functions, recommendation algorithms (hotness, product similarity, collaborative filtering), front‑end and back‑end UI, and step‑by‑step Docker deployment of MySQL, Redis, HBase, and Kafka services.
This article analyzes common causes of slow MySQL queries such as index misuse, MDL locks, and large‑table bottlenecks, then presents practical solutions like proper indexing, sharding, read/write splitting, and evaluates when to complement MySQL with ElasticSearch or HBase for better performance.
This article presents a comprehensive overview of a Flink-powered real-time recommendation system, detailing its v2.0 architecture, module functions, recommendation algorithms, front‑end and back‑end interfaces, Docker‑based deployment of MySQL, Redis, HBase, Kafka, and step‑by‑step startup procedures.
This article examines the complementary strengths of HBase and Elasticsearch, outlines three integration patterns and their associated challenges, and introduces Alibaba Cloud's Lindorm Searchindex as a SQL‑driven, low‑cost solution that simplifies storage and full‑text search for massive data workloads.
This article demonstrates how to configure Flink 1.12 to read data from HBase, covering the required environment components, HBase table creation, Maven dependencies, Java POJO and Flink‑SQL code, and showing the query results with and without printing the TableResult.
This article details a financial company's exploration of Apache Flink for real‑time processing, covering its unique business constraints, end‑to‑end data pipeline, single‑table and multi‑table use cases, implementation challenges, code snippets, data initialization, testing strategies, and lessons learned.
This article presents a comprehensive, practical guide to diagnosing and optimizing HBase read latency, covering common issues such as Full GC, region‑in‑transition, low write throughput, and high read delay, and offering client‑side, server‑side, column‑family, and HDFS tuning recommendations.
The article explores what makes code 'good'—emphasizing usability, readability, and maintainability—then details the design and implementation of a lightweight Redis‑based distributed consumption queue that alleviates HBase I/O pressure, describing its architecture, modules, logging, and performance gains.
To meet iQIYI video production’s thousands‑QPS, petabyte‑scale, frequently‑updated data and large‑table join requirements, the team built a Spark‑plus‑ClickHouse real‑time warehouse that streams Kafka changes, joins HBase dimensions, and writes to ClickHouse, reducing reporting development time from days to hours while supporting both offline and real‑time analytics.
This article explains how the HBase hbck tool verifies region consistency and table integrity, lists the most frequently used hbck commands, and describes both low‑risk and high‑risk repair procedures, including handling of RIT states and log‑based troubleshooting.
This article examines the challenges of GC pauses in low‑latency HBase services, explains ZGC’s fully concurrent architecture and key techniques such as colored pointers and read barriers, and presents experimental comparisons of ZGC and G1 GC using YCSB benchmarks, highlighting latency, throughput and CPU usage differences.
This article analyzes common causes of slow MySQL queries—especially index misuse—offers practical indexing techniques, explains MDL locks and large‑table bottlenecks, and then compares ElasticSearch and HBase as complementary solutions for high‑performance search and storage.
This article describes how the Didi HBase team tackled HBase’s weak availability and GC‑induced latency spikes by introducing a replication‑based client multi‑path read mechanism, configuring hedged reads, and adopting the Z Garbage Collector, and presents the resulting performance improvements and remaining challenges.
This article explores why MySQL queries become slow, delves into index pitfalls and optimization techniques, then compares ElasticSearch and HBase architectures, offering practical guidance on when to use each technology and how to combine them for high‑performance data retrieval.
This article explains why MySQL queries become slow, how proper index design, MDL locks, sharding, read‑write separation, and the use of ElasticSearch or HBase can improve query performance in large‑scale systems, and provides practical tips and code examples for each technique.
This article explains the background, challenges, and architectural design of a big‑data OLAP platform that integrates Apache Kylin with a BI system, detailing pre‑computation strategies, cube construction, user authentication, storage engines, and query mechanisms to achieve sub‑second analytics on massive datasets.
This article provides a comprehensive introduction to HBase, covering its origins, column‑oriented NoSQL design, storage on HDFS, logical and physical structures, read/write workflows, performance optimizations, and common interview questions for big‑data engineers.
This article explains the fundamentals and drawbacks of B+ trees, introduces the Log-Structured Merge Tree (LSM Tree) used in HBase and other NoSQL databases, and discusses how LSM architecture, Bloom filters, and MemStore improve write performance while affecting read efficiency.
This article provides a detailed, practical guide to optimizing HBase in production, covering table pre‑splitting, RowKey design, JVM memory and GC settings, MSLAB and BucketCache configuration, read‑side client and server tuning, write‑side strategies, and additional tips such as compression and scan caching.
This article details how to migrate Spark Streaming jobs to Flink Streaming by creating custom SourceFunction and SinkFunction implementations, including a RocketMQ source connector and an HBase sink, with code examples, configuration tips, and discussion of checkpointing and watermark handling.
To overcome HBase’s weak availability and GC‑induced latency spikes, the DiDi team introduced a replication‑based client multi‑read (hedged‑read) mechanism and migrated to the Z Garbage Collector, which together dramatically cut maximum and 99.9th‑percentile latencies while keeping services online during region disruptions.
This article provides a comprehensive technical comparison of HBase, Kudu, and ClickHouse—covering installation dependencies, architecture, basic read/write and query operations, real‑world use cases at Didi, a Kudu‑based real‑time data warehouse, and ClickHouse log‑analysis practices—highlighting each system’s strengths and trade‑offs for big‑data workloads.
In this talk, Yun Jin explains how HBase’s petabyte‑scale, horizontally‑scalable architecture—built on Hadoop, HMaster, RegionServers, and Zookeeper—enables e‑commerce platforms to handle extreme promotion‑day traffic by supporting high‑throughput reads/writes, time‑series monitoring, massive order storage, and robust HA, while covering essential table operations, monitoring, and troubleshooting techniques.
The article presents a comprehensive big‑data solution that stores massive GPS trajectory records in HBase, processes them with Spark to identify individuals who visited a pandemic source region, and visualizes their spatio‑temporal distribution in target cities to support precise epidemic control measures.
This article introduces HBase’s column‑oriented architecture, explains the roles of Master, RegionServer, and Zookeeper, and provides detailed environment preparation and installation commands for setting up an HBase cluster on Hadoop.
This article details how Beike's engineering team scaled Apache Kylin to handle tens of millions of daily queries by optimizing HBase configurations, reducing region count, improving data locality, addressing IO and JVM GC bottlenecks, and implementing comprehensive slow‑query detection and active‑defense mechanisms.
This article details ZTO Express's journey of adopting Apache Kylin for OLAP, comparing it with Presto, describing platform architecture, performance gains, integration with scheduling and monitoring systems, and the practical optimizations and future plans that enabled sub‑second query responses on massive daily data volumes.
This article details how a high‑traffic e‑commerce platform migrated from a single MySQL instance to a sharded architecture using Sharding‑JDBC, added Elasticsearch‑HBase for multi‑dimensional queries, and implemented zero‑downtime data migration and scaling strategies.
This article explains what HBase is, describes its Master, RegionServer, and Zookeeper components, and provides detailed environment preparation and configuration steps—including host setup, SSH key distribution, JDK installation, HBase deployment, configuration file edits, and cluster startup—so you can run HBase on a Hadoop cluster.
This article provides a comprehensive overview of Apache HBase’s architecture, detailing its core components such as RegionServer, HMaster, ZooKeeper, WAL, MemStore, and HFiles, and explains key processes including read/write paths, compaction, region splitting, load balancing, and recovery mechanisms.
The article explains HBase's RegionServer management of regions and stores, detailing HStore composition, MemStore flushing, split conditions, column family sharing within regions, and the performance implications of multiple column families, recommending a single column family design for optimal I/O efficiency.
This guide explains how to prepare a CDH‑based Spark environment for Kerberos authentication, covering prerequisite knowledge, classpath adjustments, HBase configuration files, Spark‑Env settings, user permission grants, Spark‑Submit execution, and common troubleshooting steps.
This tutorial demonstrates how to generate simulated web access logs in Python, schedule them with Crontab, collect them in real time using Flume, forward them to Kafka, process the streams with Spark Streaming, store results in HBase, and visualize the data via a Spring Boot application with ECharts.
This article presents a comprehensive guide for transferring massive MySQL datasets to HBase, covering environment setup on Ubuntu, three synchronization methods—MySQL LOAD DATA, a Kafka‑Thrift pipeline using Maxwell, and real‑time Flink processing—along with performance comparisons and practical tips for Hadoop, HBase, Kafka, Zookeeper, Phoenix, and related tools.
This comprehensive guide details three practical methods for syncing massive MySQL datasets to HBase—including Sqoop, Kafka‑Thrift, and Flink pipelines—covering environment setup, configuration, code examples, performance comparisons, and optimization tips for large‑scale data ingestion and querying.
The article details iQIYI’s migration from a Hive + MySQL OLAP stack to Apache Kylin, describing the system’s architecture, typical use cases, performance gains, independent HBase deployment, service platform for monitoring, and future plans such as automated cube building and clustering.
This article introduces Apache Phoenix, an open‑source SQL layer for HBase that enables JDBC‑based table creation, data insertion and querying, while supporting secondary indexes, transactions and various optimizations, and outlines a series covering its syntax, tools, best‑practice and real‑world use cases.
This article provides a comprehensive guide to using Sqoop for importing data from relational databases into HDFS, Hive, and HBase, as well as exporting data back to databases, covering command syntax, options, and practical examples for big‑data workflows.
This comprehensive guide walks you through setting up a pseudo‑distributed Hadoop environment, loading massive MySQL data with LOAD DATA, Python scripts, and multithreading, and then synchronizing the data to HBase using three approaches—Sqoop, a Kafka‑Thrift pipeline, and a real‑time Kafka‑Flink pipeline—while also comparing query performance of HBase and Phoenix.
This article explains Hive's role as a Hadoop‑based data warehouse, its integration with HBase, the advantages and drawbacks of that combination, introduces Apache Phoenix as a high‑performance SQL layer on HBase, and describes the open‑source NewSQL database Lealone, providing practical usage scenarios and performance comparisons.
This tutorial demonstrates how to create a VARBINARY test table in HBase using Phoenix, serialize its data with RoaringBitmap, implement a custom Spark aggregation function to merge bitmap values, and query the table via Spark SQL, showcasing a practical big-data processing workflow.
This article reviews B+ trees, introduces log‑structured merge (LSM) trees, compares their strengths and weaknesses, and explains how HBase leverages LSM trees, HFiles, compaction, and Bloom filters to achieve high‑performance storage for write‑intensive workloads.
This article explains how ZooKeeper functions as a distributed coordination service for HBase, detailing its role in master and RegionServer fault tolerance, log splitting, META table location tracking, and replication management, illustrating the underlying ZNode structures and failover mechanisms.
The article offers a detailed technical overview of OpenTSDB’s architecture and data model, explaining how it leverages HBase for scalable time‑series storage, describing core concepts, table schemas, ingestion flow, performance considerations, and future alternatives for large‑scale monitoring workloads.
This article describes JDHBase’s large‑scale KV storage, its HBase‑based replication principle, the multi‑active cluster architecture with Fox Manager, client routing, automatic failover, dynamic replication tuning, serial replication guarantees, and future directions for improving cross‑region disaster recovery.
Suning’s 618 shopping festival showcased a massive sales surge backed by its big‑data platform, which processed over 200 billion requests, handled 38.5 PB of daily data, and delivered 31.5 trillion computations, while Kafka and HBase sustained tens of millions of TPS to ensure a seamless consumer experience.
The article explains the concept of hot and cold data, why separating them reduces cost, and presents heterogeneous and homogeneous architectural solutions—including Elasticsearch, HBase, AWS S3, and cloud‑based UltraWarm—illustrated with network‑behavior and e‑commerce order system case studies.
The article details a step‑by‑step investigation of a Zookeeper outage and subsequent HBase master failure caused by an outdated Zookeeper version bug and an excessively large jute.maxbuffer setting, explaining how to identify the issue, adjust configurations, and improve region assignment performance.
This article explains HBase compaction—a key operation in the Log‑Structured Merge‑Tree model—covering minor and major compaction differences, trigger conditions, configuration parameters, selection algorithms, thread‑pool handling, and the effects on read and write performance in a big‑data database environment.
Didi's HBase team upgraded eleven clusters from version 0.98 to 1.4.8, tackling maintenance burdens and custom‑patch divergence, validating RPC and HFile compatibility, performing extensive functional and performance tests, opting for a rolling upgrade, fixing a region‑split data‑loss bug, merging critical upstream patches, and establishing a reusable migration methodology.
This article explains how to configure JVM memory, choose appropriate garbage‑collector settings, tune HBase region split policies, optimize BlockCache implementations, and select suitable compaction strategies to improve HBase performance on clusters of various sizes.
The article explains how HBase tracks each region's lifecycle states in hbase:meta and ZooKeeper, lists all possible states with their color codes, and describes the master‑region server interactions for opening, closing, splitting, and merging regions.
This article explains how HBase uses WAL and MemStore to create HFiles, describes the two compaction types (Minor and Major), and provides detailed recommendations for tuning key compaction-related configuration parameters to improve query performance and reduce HDFS impact.
This tutorial demonstrates how to rapidly set up and use HBase inside a Docker container, covering Docker installation, image pulling, container execution, host configuration, accessing the HBase Web UI and shell, Zookeeper interaction, and a Java API example for beginners.
YouZan’s real‑time loss‑prevention platform monitors database binlogs, transforms and verifies transaction data across five loosely coupled layers, handling 200 million daily messages and 60 million checks with dynamic sharding, caching and distributed locks to detect over‑charges, duplicate refunds, migration inconsistencies and unauthorized data changes.
This article summarizes the design, architecture, and key modules of a company-wide data lake platform—named “Tianchi”—built on HBase, Kafka, and Elasticsearch, detailing data ingestion, strategy output, metadata management, indexing, monitoring, and offline analysis, and shares lessons learned and future plans.
This article categorizes and explains the most important HBase 1.1.2 configuration parameters—covering Region sizing, BlockCache strategies, Memstore thresholds, Compaction behavior, HLog handling, Call Queue tuning, and miscellaneous settings—while offering practical recommendations for optimal production deployment.
This article provides a comprehensive introduction to HBase, covering its origins, key characteristics, architecture components, installation steps, basic shell commands for table management, data structures, read/write processes, and high‑availability configuration within the Hadoop ecosystem.
The article analyzes HBase's CP‑oriented CAP limitations, proposes native AP support via Replica, decouples WAL IO to Kafka, optimizes MTTR, introduces multi‑datacenter active/active disaster recovery, and redesigns client write paths and LogSplit processing for higher availability and throughput.
The article explains how maintaining an optimal number of HBase regions (typically 20‑200 per RegionServer) and appropriate region size, along with careful MemStore and compaction settings, can prevent memory pressure, reduce GC pauses, and enhance overall cluster stability and throughput.
This article details a series of practical enhancements applied to an HBase 1.2.4‑based cluster—including layered BucketCache, data pre‑heating, GC‑friendly object pooling, and a multiget concurrency model—that together raise throughput several‑fold and consistently keep P99 latency below 50 ms in YCSB benchmarks.
The article systematically explains how HBase's MemStore uses a SkipList‑based model and introduces successive memory‑management optimizations—MemStoreLAB, ChunkPool, off‑heap chunks, CompactingMemStore and the CCSMap data structure—to reduce object overhead, GC pressure and improve throughput.
This article explains the HBase data‑write process—including WAL logging, MemStore caching, and HFile flushing—identifies three levels of write‑blocking (HFile, MemStore, RegionServer), and provides configuration tweaks to mitigate blocking in production environments.
This article examines Pharos, Everbright Bank's home‑grown HBase indexing middleware, detailing why existing secondary‑index solutions fall short, the design goals of low latency, simple architecture and non‑intrusiveness, and the concrete storage, pagination, and transaction‑consistency techniques that enable fast complex queries on massive data.
This article details the evolution of HBase cluster deployment from mixed‑hardware/software setups to fully independent clusters, explains hardware and software considerations, presents memory and region planning, outlines key configuration parameters, and provides Spark integration examples for batch and real‑time queries and writes.
This guide explains how to optimize HBase performance by adjusting JVM memory settings, selecting appropriate garbage collectors, configuring MSLAB and in‑memory compaction, choosing region split policies, tuning BlockCache implementations, and applying suitable compaction policies for different workloads.
The article explains how to design HBase RowKeys by dispersing keys, controlling their length, and ensuring uniqueness, providing concrete techniques such as salting, hashing, reversing values, and a practical example with table creation to improve scan performance and prevent region hotspot issues.
This article presents a comprehensive guide for synchronizing massive MySQL datasets to HBase, covering environment preparation, fast MySQL data loading techniques, and three practical pipelines—Sqoop, Kafka‑Thrift, and Kafka‑Flink—along with performance comparisons and optimization tips for large‑scale data processing.
This article compiles a series of HBase questions and answers covering write performance, bulk loading, single‑node configuration, column scalability, transaction isolation, fast deletion methods, off‑heap optimizations, bulkload modes, Hive integration, direct HFile reads, and region planning.
This article explains how to optimize HBase performance by properly configuring JVM memory, selecting suitable garbage‑collection strategies, enabling MSLAB and BucketCache, and adjusting read/write cache ratios to reduce fragmentation and improve throughput.
This article explains the core mechanisms of HBase—Flush and Compaction—detailing why they are needed, the conditions that trigger Flush, the types and triggers of Compaction, and provides practical recommendations for tuning the most important configuration parameters to improve write and read performance.
This article explains ZooKeeper’s architecture, key concepts such as roles, sessions, ZNodes, versioning, ACLs, and watchers, and demonstrates how it powers essential big‑data components like Hadoop’s ResourceManager and HBase’s master election, naming service, and distributed locking.
This article explains how to use HBase's built‑in PerformanceEvaluation tool to run baseline read/write latency tests (P99 and P999), describes key command‑line parameters, presents benchmark results for random and sequential operations, and discusses the implications for HBase performance tuning.
This article provides a comprehensive overview of HBase, covering its LSM‑Tree based storage engine, key‑value data model, column‑family storage design, indexing mechanisms, major advantages and drawbacks, and typical scenarios where HBase excels for massive, high‑throughput data workloads.
This article introduces OpenTSDB, a highly scalable time‑series monitoring system built on HBase, explains its architecture, demonstrates how it solves common monitoring challenges, and shows practical usage examples including data modeling, collector integration, and real‑world deployment insights.
This article explains how to integrate Apache Phoenix with CDH‑based HBase, covering Phoenix overview, version selection, parcel installation, HBase configuration, command‑line usage, mapping existing tables, creating schemas and views, building secondary indexes, and comparing different index types for performance optimization.
JDHBase, JD.com’s large‑scale KV store, powers billions of daily reads and writes across 7,000 nodes, and this article details its multi‑active, cross‑region architecture—including HBase replication fundamentals, Fox Manager routing, automatic failover policies, dynamic replication tuning, and serial replication to ensure strong consistency.
This article summarizes NetEase Hangzhou Research Institute expert Fan Xinxin's presentation on HBase, covering its role in the big‑data ecosystem, core production scenarios, RIT and HBCK troubleshooting techniques, and systematic monitoring and log‑analysis methods for diagnosing HBase issues.
This article examines the design of a real-time data warehouse built on Flink, Kafka, and HBase, compares it with traditional offline warehouses, and discusses key challenges such as data accuracy, latency, and the complexity of maintaining real-time dimension tables.
This article presents a step‑by‑step guide for building a real‑time data pipeline using Kafka as a message buffer, Spark‑Streaming's Direct Approach for processing, and HBase for storage, including code examples, Maven configuration, local cluster setup, and troubleshooting tips.
This article describes JDHBase’s large‑scale KV storage deployment, its HBase‑based asynchronous replication mechanism, the multi‑active architecture with active‑standby clusters, client interaction via Fox Manager, automatic failover strategies, dynamic replication tuning, and serial replication techniques to ensure data consistency across data centers.
This article introduces Apache Kylin, details its deployment at Tongcheng Yilong, explains the design of a large‑scale trajectory model, and provides step‑by‑step optimization techniques—including cube dimension reduction, HBase rowkey tuning, build parameter tweaks, high‑cardinality handling, and query compression disabling—to achieve sub‑second OLAP queries on multi‑terabyte data.
Youzan’s evolution of HBase bulk‑load—from manual MapReduce jobs to Hive‑SQL and finally Spark—demonstrates how generating HFiles on HDFS, partitioning by region, sorting keys, and handling serialization issues enables billions of records to be loaded efficiently without disrupting production clusters.
This article reviews Alibaba's decade‑long evolution of HBase high‑availability, covering large‑cluster design, MTTF/MTTR metrics, disaster‑recovery strategies, traffic switching, and performance optimizations that together enable millions of requests per second with near‑zero downtime.
This article explains how to create custom Flink SourceFunction and SinkFunction implementations, demonstrates a RocketMQ source and an HBase sink with full code examples, and discusses checkpointing, event‑time handling, and deployment of the streaming job on a Flink‑on‑YARN cluster.
This article explains the fundamentals of the CMS (Concurrent Mark Sweep) garbage collector, presents a comprehensive set of JVM parameters optimized for HBase clusters, and provides detailed analysis of key settings to improve performance and reduce GC pauses.
This tutorial walks through setting up the Python happybase library, installing JPype for Java integration, and demonstrates end‑to‑end code examples for connecting to an HBase Thrift server, generating row keys via Java utilities, querying data, and handling type conversions.
A recent data platform salon in Beijing gathered five leading experts who shared practical knowledge on data middle platforms, distributed transaction patterns, SQL audit design, Amazon Aurora's architecture, and JD's large‑scale HBase deployment, offering actionable guidance for modern enterprise data engineering.
This article outlines the challenges of various big‑data scenarios such as financial risk control, recommendation systems, and social feeds, explains why Spark is chosen over alternatives, describes a one‑stop data platform architecture with Spark‑HBase integration, and shares best‑practice tips and case studies.
At a Tsinghua University forum, Hulu presented a comprehensive overview of its big‑data solutions for advertising and streaming, covering challenges of massive, complex data, the limits of MySQL, and advanced techniques using HBase, Protobuf, Redis batch pipelines, and its own MPP engine Nesto for high‑performance, scalable analytics.
This article explains how to reverse the typical data flow by extracting massive Rowkeys from HBase with MapReduce, storing them on HDFS, and then using batch Get operations to retrieve the full records and write them into Kafka, while handling retries and monitoring progress.
Youzan’s personalized recommendation platform combines a four‑layer architecture—data, storage, service, and application—with multi‑dimensional real‑time, offline, and cold‑start recall algorithms, Wide&Deep ranking, HBase/Druid storage, and configurable scene strategies to boost user conversion, traffic monetization, and future scalability.
