Youzan’s personalized recommendation platform combines a four‑layer architecture—data, storage, service, and application—with multi‑dimensional real‑time, offline, and cold‑start recall algorithms, Wide&Deep ranking, HBase/Druid storage, and configurable scene strategies to boost user conversion, traffic monetization, and future scalability.
This article reviews Alibaba HBase's evolution toward high availability, covering large‑cluster architecture, reliability metrics (MTTF/MTTR), disaster‑recovery strategies such as data replication and traffic switching, performance optimizations for extreme latency requirements, and lessons learned for building resilient distributed database services.
This article explains HBase's data model and key components, details column descriptor options such as BloomFilter, Compression, Versions, TTL, and MinVersion, and provides practical design guidelines for columns, rowkeys, high vs. wide tables, region pre‑splitting, and hotspot mitigation to achieve optimal performance.
This article explains HBase’s data model and provides comprehensive table‑design strategies—including column‑descriptor options, row‑key best practices, high‑vs‑wide table trade‑offs, region splitting and pre‑splitting techniques—to help achieve optimal performance and scalability in large‑scale NoSQL workloads.
This article explains how Apache Kylin, an open‑source distributed analytics engine built on Hadoop/Spark, achieves sub‑second OLAP query performance through pre‑computed cubes, a layered cuboid generation algorithm, bitmap‑based distinct counting, dimension optimization techniques, and tight integration with HBase for storage and fast SQL querying.
This article explains how Apache Phoenix implements global secondary indexes using separate HBase tables, demonstrates index creation and data synchronization with example SQL, and provides design guidelines to optimize query latency and avoid full‑table scans in big‑data environments.
This article analyzes the business requirements, technical challenges, and mainstream architectural solutions for large‑scale feed streams, and proposes a step‑by‑step evolution path—from a simple push model using cloud Kafka and HBase to hybrid push‑pull and recommendation‑driven designs—suitable for startups and rapidly growing platforms.
This article explains how Alibaba Cloud's BDS migration service enables continuous, high‑performance migration of HBase clusters—including schema, full data, and incremental sync—across version upgrades, hardware changes, network migrations, and cross‑region scenarios, while ensuring stability and minimal impact on live workloads.
The article explains that HBase and Hive occupy distinct roles in big‑data systems—HBase handles real‑time random queries on massive detail data, while Hive provides batch‑oriented SQL‑based processing on HDFS—and describes how they are typically combined in a data pipeline.
This article introduces Apache Phoenix as an open‑source SQL layer for HBase, explains its core concepts and performance benefits, provides step‑by‑step installation instructions, demonstrates basic SQL operations and shows how to use the Phoenix Java API with code examples.
The article explains why HBase client connections should not be pooled, describes common misuse patterns, and details how the heavyweight, thread‑safe Connection object internally manages connections to HMaster, RegionServers, and ZooKeeper, recommending a single shared Connection per application.
This comprehensive guide introduces Apache Phoenix as an HBase SQL layer, covering quick‑start steps, supported data types, DML syntax, salted tables to prevent hotspots, various secondary index types, bulk‑load methods, auto‑increment IDs, dynamic columns, pagination, query plan analysis, and data migration techniques.
This article details the design, architecture, and operational experience of XStore, a Hadoop‑backed sample storage system that handles billions of APK and other binary samples, addressing functional and non‑functional requirements such as real‑time upload, large‑scale storage, high‑performance reads, and disaster recovery.
This article provides a comprehensive, illustrated explanation of Apache HBase's architecture, covering its master‑slave components, region management, Zookeeper coordination, data flow for reads and writes, storage structures, compaction processes, fault recovery, and the system's strengths and limitations within the Hadoop ecosystem.
This article provides an in‑depth overview of Apache Kylin’s pre‑computation architecture, data‑warehouse concepts, step‑by‑step cube creation from Hive tables, and advanced optimization techniques such as derived dimensions, aggregation groups, and HBase row‑key encoding to achieve sub‑second OLAP queries on massive datasets.
This article explains Zookeeper’s core concepts, its ZAB consensus protocol, and surveys its essential roles in major big‑data components such as Kafka, Hadoop, HBase, and Solr, illustrating how it provides configuration, naming, coordination, leader election, and high‑availability services across distributed architectures.
This article explains how 360 Search processes billions of web pages daily, detailing its backend architecture, offline indexing, online retrieval, index organization, and relevance models that enable efficient search over a hundred‑billion‑scale web corpus.
This article provides a comprehensive overview of HBase, covering its distributed column‑oriented architecture, data model components, storage mechanisms, read/write processes, WAL lifecycle, MemStore flushing, region splitting and merging, and failure recovery within the Hadoop ecosystem.
This article examines JD.com's massive HBase deployment, detailing its evolution from early adoption to a 90PB, 7,000‑node cluster, the platform's architecture, multi‑active disaster recovery, multi‑tenant isolation, and the integration of Phoenix for SQL‑based access, offering practical insights for large‑scale distributed storage.
This article describes how Kuaishou built BitBase on HBase to store and compute billions of user feature logs with millisecond‑level latency, covering business requirements, technical selection, bitmap data modeling, system architecture, device‑ID handling, performance results, and future roadmap.
The article describes how a large‑scale ETL process that previously used HBaseStorageHandler caused severe region server pressure, and how a new HBase‑to‑Hive task based on SnapshotScanMR was designed to bypass region servers, halve execution time, and double scanning performance.
This article explains the complete HBase write flow, the problems caused by rapid writes such as memstore overflow and RegionTooBusyException, and provides configuration and operational strategies to avoid RegionServer out‑of‑memory crashes.
This article explains how leveraging HBase's SnapshotScanMR feature to create a custom hbase2hiveBySnapshot task dramatically reduces region server pressure, halves ETL execution time, and improves cluster stability for large‑scale data back‑fill operations.
This article provides a detailed technical analysis of Phoenix's native index creation and maintenance mechanisms, the underlying source code for index building, the role of coprocessors, and the complete SQL compilation pipeline from parsing to execution, highlighting how hints and optimizers influence index usage.
The article explains why sharding (splitting databases and tables) is the preferred solution for handling massive user, order, and transaction data in high‑traffic internet applications, comparing it with partitioning and NoSQL/NewSQL alternatives, and detailing practical middleware choices, sharding column selection, and integration with Elasticsearch and HBase.
The article explains how excessive HBase partitions can cause frequent flushes, compaction storms, high memory usage, long master assignment times, and reduced MapReduce concurrency, and provides formulas and guidelines for calculating a reasonable number of regions per RegionServer.
The article presents Alibaba Cloud's real-time computing solution based on Flink and HBase, covering market competition, open‑source ecosystem, containerized architecture on Kubernetes, and typical applications such as online education video analysis, city‑brain traffic management, and fraud detection.
Apache Phoenix is an open‑source SQL layer for HBase that lets developers use standard JDBC instead of the native HBase client API, offering features such as secondary indexes, transactions, and various SQL‑level optimizations while supporting full table creation, insertion, and querying.
This article explains HBase's compaction mechanism, covering why it is needed, the differences between minor and major compaction, the conditions that trigger compaction, key configuration parameters, thread‑pool handling, compaction policies, and how compaction influences read and write performance in a large‑scale NoSQL database.
This article presents a detailed post‑mortem of a 10‑node HBase cluster failure caused by excessive region count and memstore pressure, analyzes HDFS and datanode log errors, and outlines configuration adjustments and operational recommendations that restored the service and prevented future outages.
The article analyzes how HBase regions being assigned to a single RegionServer create reliability issues such as jitter, service interruptions, and data loss, examines the underlying hardware, OS, and operational factors, and proposes system optimizations and replica-based high‑availability strategies to mitigate these problems.
The article explains why HBase read operations are complex due to its LSM‑Tree storage and multi‑version design, details the step‑by‑step Scan workflow, discusses the reasons for its multi‑request architecture, compares scan performance with Parquet and Kudu, and offers recommendations for large‑scale data scanning.
This article details Didi's use of HBase for various online and offline workloads, covering multi‑language support, data types, rowkey designs for order, trajectory and ETA scenarios, multi‑tenant resource management with DHS and RS Group, and operational best practices.
This article explains the fundamentals of NoSQL databases, the CAP theorem trade‑offs, and why HBase—a column‑oriented, distributed NoSQL system—offers strong consistency, automatic sharding, high availability, and seamless Hadoop integration, while also outlining its ideal scenarios and limitations.
This article introduces HBase, its origins from Google Bigtable, core concepts such as RowKey, Column Family, and Versioning, and explains its logical and physical table views, storage mechanisms, and cluster architecture within the Hadoop ecosystem.
YouZan’s order data synchronization moves changes from MySQL through Canal‑parsed binlogs into a message queue, then uses sequential SeqNo‑based optimistic locking and HBase’s column‑version timestamps to guarantee ordering for both single‑ and multi‑table updates, while a Logstash‑style configurable pipeline feeds ES for search and HBase for detail queries, eliminating ordered‑queue bottlenecks and ensuring high‑throughput consistency.
This article details Youzan's evolution from a simple Flume‑based log collector to a multi‑tenant, Kafka‑buffered, Spark‑processed, HBase‑backed logging architecture that now handles hundreds of billions of log entries per day, highlighting challenges, design decisions, and future improvements.
This guide introduces Hadoop and HBase fundamentals, explains their architectures and advantages, and provides step‑by‑step instructions for setting up a multi‑node Hadoop cluster, configuring core services, installing HBase, and performing basic HBase shell operations.
This article explains how JD.com leverages the open‑source HBase database for massive, low‑latency data storage across various business lines, detailing its architecture, multi‑tenant isolation, disaster‑recovery mechanisms, and integration with Phoenix SQL for OLTP workloads.
This article introduces OpenTSDB as a distributed, scalable time‑series database built on HBase, explains its architecture, data model, and storage optimizations, presents real‑world monitoring use cases, analyzes performance issues caused by high‑cardinality tags, and details the solution steps taken to restore query speed.
This article provides a comprehensive technical guide on HBase, covering its core concepts, advantages and drawbacks, architecture layers, practical use cases, and a detailed step‑by‑step process for large‑scale cross‑datacenter migration using snapshot‑based strategies, with commands, diagrams, and lessons learned.
This article explains the differences between row‑oriented and column‑oriented storage, compares their trade‑offs, and introduces HBase’s column‑family architecture, including row keys, column qualifiers, timestamps, cells, and how it maps to a multi‑dimensional map structure.
The article first outlines HBase’s overall architecture and core components, then details the end‑to‑end read path—from client request routing to RegionServer processing, data organization and filtering—and finally presents practical client‑ and server‑side optimizations such as heterogeneous storage, HDFS short‑circuit, hedged reads, high‑availability reads, and warm‑up failure fixes, illustrated with Youzan’s production cluster.
This article explains HBase as a high‑performance, column‑oriented distributed storage system, outlines its advantages and limitations, presents real‑world scenarios such as seller operation logs and message logs, and details its data structures, architecture components, and design considerations for big‑data applications.
The article summarizes the technical talk from the 58 Group technology salon, detailing the architectures, scalability, high‑availability mechanisms, and storage models of Kuaishou's BlobStore and 58's WOS, and compares their design choices for large‑scale object storage.
This article introduces Kylin, a HBase‑based multidimensional analysis platform, explains its architecture and various performance optimizations—including multi‑tenant support, dimension dictionary handling, and cube size estimation—while showcasing a real‑world deployment and case study at 58.com.
This article analytically breaks down HBase's write pipeline, quantifies disk and network overheads for massive random writes, derives formulas for resource consumption under realistic assumptions, and offers concrete tuning recommendations to optimize throughput and reduce cost.
This article introduces HBase's ecosystem, including its components like OpenTSDB for time-series data, Kylin for cube analysis, Phoenix for SQL operations, and GeoMesa for spatial data, along with the author's experience in deploying these in a production environment.
This article details the architecture and operational enhancements of 58.com’s HBase platform, covering multi‑tenant resource isolation, various data access APIs, bulk import/export mechanisms, and a series of performance optimizations that improve stability and scalability for massive data workloads.
This article examines the challenges of Youzan's order export system, describes the migration from PHP‑based scripts to an Elasticsearch and HBase stack, and details the step‑by‑step configuration‑driven refactor—including enum field definitions, Groovy scripts, strategy patterns, plugin architecture, and quality‑assurance practices—that enabled million‑order exports with high performance and stability.
This article introduces HBase fundamentals, explores effective RowKey and secondary index design principles, discusses demand analysis, presents techniques such as reversing, salting, hashing, and reviews real-world case studies for OpenTSDB, JanusGraph, and GeoMesa, offering practical guidance for scalable NoSQL data modeling.
This article provides a comprehensive overview of Apache HBase, covering its core architecture, key features such as automatic sharding, LSM‑Tree storage, separation of storage and compute, the ecosystem, real‑world use cases, recent 2.0 enhancements, upgrade guidance, future plans, and community recruitment information.
This article details the practical performance‑tuning steps applied to a large‑scale HBase deployment handling 80 billion daily log entries, covering rowkey redesign, region redistribution, HDFS write‑timeout fixes, network‑topology adjustments, and JVM parameter tweaks that together stabilized the system and dramatically improved throughput.
This article walks through a real‑world incident where sudden traffic drops were traced to abnormal TCP states—TIME_WAIT, CLOSE_WAIT, and SYN_RECV—by examining monitoring data, explaining the TCP handshake, reviewing relevant kernel parameters, and debugging Java/ZooKeeper/HBase code to identify and fix the root cause.
This article summarizes the current usage of HBase at Ping An Technology, the challenges it addresses, detailed client‑ and server‑side performance and stability optimizations, high‑availability mechanisms, data migration strategies, monitoring and repair practices, and future development plans.
The article details the evolution from an offline to a real‑time HBase data warehouse, covering business scenarios, the use of Maxwell for MySQL‑to‑Kafka ingestion, Phoenix for SQL access, CDH cluster tuning, monitoring, and several production case studies.
This article summarizes Guo Dongdong’s presentation on leveraging HBase for a risk‑control platform, detailing its architecture, data import/export mechanisms, indexing, region server recovery challenges, monitoring, SQL interception, dual‑cluster high‑availability, and future enhancements for large‑scale, low‑latency big‑data services.
This article presents NetEase's extensive use of HBase within its big‑data platform, covering the system’s role, real‑world application scenarios, common RIT issues, HBCK repair methods, and a systematic approach to monitoring and troubleshooting performance problems.
This article explains how Alibaba's search offline platform combines massive batch and real‑time processing, leveraging Hadoop, HBase, Flink (Blink) and a unified full‑incremental model to handle tens of billions of daily records and millions of TPS for diverse business lines.
This article introduces Hadoop’s open‑source big‑data framework, explains its core components HDFS and MapReduce, and outlines four key advantages—ease of deployment, robustness, scalability, and simplicity—while also covering HBase as the Hadoop‑based column‑oriented database.
This article details a big‑data precise‑marketing solution that leverages HBase storage and Roaring BitMap indexing to efficiently handle billions of user records, describing project background, technology selection, architecture, partitioning strategy, and coprocessor implementation for fast multidimensional queries.
This guide details how to construct a real-time data processing platform on CentOS 7 using the Hadoop ecosystem—installing and configuring Zookeeper, Maven, Hadoop, Kafka, HBase, Spark, and Flume—followed by a Spark Streaming job that consumes Kafka messages and writes them into HBase.
This case study details how Beijing 恒远华信息技术有限公司 built a dynamic face‑capture and real‑time recognition solution on Huawei FusionInsight HD, leveraging deep‑learning algorithms, distributed storage, and stream processing to handle hundreds of millions of faces with high speed, efficiency, and security.
The article explains how relational databases struggle with many‑to‑many and deep relationship queries, compares popular graph databases, details Titan's modular architecture, data model, Gremlin query examples, storage layout, and demonstrates its successful deployment at Paipaidai for large‑scale fraud detection, achieving over 25% efficiency gains.
This guide introduces the most commonly used HBase operational tools—including Canary, hbck, HFile viewer, CopyTable, Export/Import, ImportTsv, CompleteBulkload, RowCounter, CellCounter, and clean utilities—explaining their purposes, typical use‑cases, and exact command syntax for effective cluster administration.
This article introduces Hulu's in‑house OLAP engine Nesto, detailing its near‑real‑time data ingestion, nested data model, TB‑level storage using HBase and Parquet, MPP query execution, custom predicate library, and the overall architecture that enables sub‑second ad‑hoc queries for user analytics.
This article explains how Alibaba engineers reduced young‑generation garbage‑collection pauses in large‑scale HBase deployments from over a hundred milliseconds to just a few milliseconds by redesigning memory management with CCSMap, BucketCache, and the tenant‑aware AliGC algorithm.
This article explains how HBase achieves scalable region auto‑splitting, detailing the various split policies, the algorithm for locating split points, the transactional split workflow, reference file handling, data migration via compaction, cleanup procedures, and common troubleshooting tips.
The article uses everyday observations from a McDonald’s restaurant to illustrate core distributed system concepts such as master‑slave architecture, two‑phase commit, microservice decomposition, task queues, and container orchestration, showing how these principles apply to backend engineering.
This article explains the design and implementation of a unified seller‑operation logging platform that uses Kafka for ingestion, Storm for real‑time processing, Elasticsearch for hot‑data search, and HBase for cold‑data storage, detailing the challenges faced and the optimizations applied.
This article details JD's end‑to‑end seller log system architecture, explaining why Kafka, Storm, Elasticsearch and HBase were chosen, the challenges faced during scaling, and the practical solutions implemented to achieve a unified, high‑throughput logging platform for merchants and operations.
This article explains the design of a real-time customer behavior collection platform that uses NSQ for messaging, Storm for streaming processing, and HBase for storage, covering architecture, data flow, reliability guarantees, and deployment considerations.
At HBaseCon 2017 Asia, Alibaba showcased a series of groundbreaking HBase enhancements—including strong synchronous replication, SQL-on-HBase capabilities, cross‑cluster range data copy, and read/write path optimizations—that dramatically improve performance, reliability, and usability for large‑scale big‑data storage.
This article details Taobao’s multi‑layer massive data platform, covering its five‑tier architecture, the 1500‑node Hadoop “Cloud Ladder” for batch processing, the low‑latency “Galaxy” stream engine, MySQL‑based MyFOX, HBase‑based Prom storage, the glider middle‑layer, and sophisticated caching strategies that together support petabytes of data and millions of daily queries.
This article explains how Didi leverages HBase across multiple business scenarios—including order lifecycle queries, driver‑passenger trajectory tracking, ETA calculations, and cluster monitoring—while addressing multi‑language support, rowkey design, GeoHash indexing, and multi‑tenant resource management.
Facing the challenge of storing over 120 billion daily monitoring points from hundreds of thousands of servers, Tencent’s monitoring platform migrated from a custom solution and OpenTSDB to a finely tuned HBase architecture, achieving 3‑5× higher throughput, improved reliability, and significant storage savings.
This article analyzes the limitations of the original TMP monitoring storage architecture, evaluates OpenTSDB's shortcomings at large scale, and details the design, implementation, and performance tuning of a custom HBase-based solution that achieves 3‑5× higher throughput for billions of monitoring data points per day.
This article explains how Didi leverages HBase’s distributed architecture, multi‑language APIs, and custom rowkey designs to support online order queries, driver‑passenger trajectory tracking with GeoHash, real‑time ETA calculations, and a monitoring platform, while managing multi‑tenant resources through DHS and RS Group.
This guide details the step‑by‑step process for replacing a faulty ZooKeeper node (myid 5) in a five‑node cluster, covering configuration updates in zoo.cfg, Hadoop’s hdfs‑site.xml, yarn‑site.xml, HBase‑site.xml, and the required service restarts to ensure continuous high‑availability.
This article introduces ZooKeeper’s fundamental architecture, explains its key concepts such as cluster roles, sessions, ZNodes, watches, and ACLs, and then details how it powers essential distributed coordination tasks—including configuration management, naming services, master election, and distributed locks—in large‑scale Hadoop and HBase ecosystems.
This article explains why Spark Streaming combined with Kafka can only guarantee at‑least‑once delivery, outlines the challenges of delayed and out‑of‑order events, and presents practical offline‑repair, deduplication, and output‑format techniques—including code examples—to achieve exact‑once semantics in big‑data pipelines.
This article details how Alibaba has evolved HBase from an internal storage solution to a cloud service, covering its architecture, high‑availability design, asynchronous and synchronous replication, multi‑link data flows, cost‑effective redundancy, performance optimizations, and future development directions.
This article explains how Alibaba built, customized, and operated a massive HBase platform—covering its architecture, high‑availability design, asynchronous and synchronous replication, multi‑link data flow, cost‑aware redundancy, cross‑cluster migration, performance optimizations, and future directions for the distributed NoSQL database.
The migration of 500 GB HBase and 5 TB Solr data from version 0.94.15 to 1.2.1 required fixing hardware clock drift, DNS hostname issues, and missing Snappy support, and demonstrated that a brute‑force HDFS transfer is more reliable than import/export when handling deprecated parameters.
Meituan Waimai’s Feature Archive platform processes billions of daily orders by managing ~200 user and 400 merchant tags through a three‑layer architecture—Hive, Elasticsearch, HBase, and MySQL—offering visual tag selection, instant self‑service queries, full data extraction, and a predicate‑logic query language, while supporting future extensibility.
This article reviews major big‑data storage and query architectures—including HBase, Dremel/Parquet, pre‑aggregation systems, Lucene, and the custom Tindex solution—evaluating their strengths, weaknesses, and suitability for real‑time, high‑volume analytical workloads.
Facing the challenges of searching across 60 million users and 1.3 billion products, Weidian’s engineering team designed a dump‑based indexing pipeline—Ergate—that consolidates, transforms, version‑controls, and monitors data from MySQL to HBase, enabling fast, flexible, and reliable search across massive datasets.
The article explains HBase’s write‑ahead‑log architecture, detailing how client put/delete requests travel through RPC to the RegionServer, are processed by MultiRowMutationService, written to the WAL via FSHLog.append and sync, and finally stored in MemStore, while describing durability options and the underlying source‑code call chain.
This article summarizes the most recent releases of major NoSQL and NewSQL databases—including MongoDB 3.4, Redis 3.0/3.2, HBase 1.2.4, Apache Geode 1.0, TiDB RC1, and the final update of RethinkDB—highlighting key new capabilities, performance improvements, and operational enhancements.
This article examines the common causes of high read latency in HBase—such as full GC, region‑server imbalance, low write throughput, and inefficient client settings—and provides concrete optimization steps for the client, server, column‑family design, and HDFS layers to dramatically improve performance.
The article explains Taobao’s five‑layer data product architecture—covering data sources, compute, storage, query, and product layers—and describes how massive volumes of data are ingested, processed in batch and streaming, stored in MySQL and HBase clusters, and served efficiently through a unified middle‑layer and sophisticated caching mechanisms.
Phoenix adds a SQL layer to HBase, enabling easy table creation, data import, and complex queries via JDBC, with features like secondary indexes and integration with Spark, Hive, and more, illustrated through step‑by‑step examples and sample code.
This article explains how HBase finds the correct region server for a given row key using the hbase:meta table stored in ZooKeeper, and describes the write path involving MemStore, HLog, StoreFile creation, and subsequent maintenance tasks.
HBase stores huge tables by splitting them into regions, distributing these across region servers managed by a master, and further dividing each region into column-family stores, memstores, and StoreFiles, forming a layered architecture built on Hadoop’s distributed storage.
This article compares traditional relational table design with HBase's column‑family model, showing how tables, rows, column families, and versioned cells are defined and how data is inserted using HBase's put commands.
The article describes how YHD's precision recommendation team built a real‑time, highly available, traceable general recommendation platform, detailing its background, overall architecture, visual configuration and traceability subsystems, and reporting significant improvements in development speed, reuse and user satisfaction.
This article presents a comprehensive overview of the Qdata session on OLAP engine exploration, detailing the limitations of traditional MySQL‑based solutions, the requirements for large‑scale analytics, the architecture and theoretical foundations of Apache Kylin, its cube construction process, storage in HBase, query rewriting, real‑world flight‑ticket data applications, and the encountered challenges with corresponding optimization practices.
The article presents a method that leverages HBase to capture, store, index, and quickly retrieve massive network packets, using PF_RING and libpcap for high‑performance capture and providing APIs for time‑, IP‑, protocol‑, and port‑based packet backtracking.
This article presents Qunar's experience transitioning from MySQL‑based OLAP to Apache Kylin, detailing the performance challenges, required features, Kylin's architecture and theory, cube construction process, storage mechanisms, real‑world applications, and the pitfalls and optimization practices discovered along the way.