The article explains how partitioning splits tables within a single database and how sharding distributes data across multiple database instances, comparing their types, advantages, limitations, and trade‑offs, and provides practical examples and a decision framework for choosing the right strategy.
In a SaaS analytics platform with a 45‑billion‑row, 1.2 TB events table, the author shows how moving to native PostgreSQL range partitioning by event_time, combined with zero‑downtime migration steps, reduced the typical P99 query latency from 43 seconds to about 0.3 seconds while enabling easier data lifecycle management.
In PostgreSQL versions prior to 14, a DELETE without a partition filter on a table with 20,000 partitions caused the server to allocate over 120 GB of memory within 60 seconds; the article shows how to reproduce the issue, diagnose it with MemoryContextStat and perf, and explains the fix introduced in PG14.
This article details production-ready Spring Batch architecture for massive data migration, covering core Job/Step/Chunk model, partitioning for parallelism, streaming file reads, JDBC batch tuning, retry/skip fault tolerance, checkpoint restart via BATCH_* tables, memory/cursor trade-offs, dynamic chunk sizing, monitoring metrics, and scheduler-executor decoupling with XXL-JOB or Airflow.
This guide walks through Linux disk management fundamentals, covering how to partition disks with fdisk or gdisk, create filesystems using mkfs, mount and unmount devices, configure automatic mounts via /etc/fstab, and handle special cases such as ISO images and Windows shares.
Kafka’s exceptionally high throughput stems from a combination of design choices—including sequential disk writes, zero‑copy data transfer, batch processing, partition‑level parallelism, OS page‑cache utilization, and producer‑side compression—that together minimize I/O overhead, CPU usage, and latency.
The article explains that message ordering is a spectrum—from strict global ordering to partition-level and finally business-key ordering—and examines the trade‑offs, pitfalls, and engineering techniques needed to maintain order while scaling to tens of millions of QPS.
The article walks through four generations of partition‑selection strategies—from naive random and round‑robin to key‑hash, sticky, and finally intelligent load‑aware routing—explaining how hotspot keys, batch inefficiency, consumer skew, and fault amplification threaten stability at 10 M QPS and offering concrete engineering actions to design, test, monitor, and switch strategies in large‑scale message‑queue deployments.
This article examines why message ordering is critical in distributed systems, explains how partition mechanisms in Kafka, RocketMQ, and RabbitMQ enable ordered consumption, and provides detailed SpringBoot implementations, best‑practice guidelines, partition‑key design principles, concurrency settings, idempotency, and real‑world case studies to ensure reliable sequential processing.
This step‑by‑step guide walks Linux newcomers through choosing a distribution, creating a bootable USB, partitioning, BIOS setup, installation pitfalls, and post‑install tweaks so they can avoid common traps and get a working system on the first try.
This article presents ten practical MySQL advanced techniques—including EXPLAIN analysis, composite and covering indexes, index condition push‑down, prefix indexes, window functions, CTEs, JSON handling, partition tables, join optimizations, variables, online DDL, and generated columns—each illustrated with real‑world SQL examples and execution‑plan screenshots to boost efficiency in high‑volume, high‑concurrency environments.
This article breaks down Kafka's high‑throughput design by detailing how partitioning enables parallelism, sequential disk writes and zero‑copy reduce I/O overhead, and OS page cache maximizes memory efficiency, together allowing millions of messages per second.
This article explains how Kafka’s sequential disk writes, zero‑copy data path, partition‑based parallelism, and configurable broker and partition settings enable linear‑scale throughput that can reach millions of transactions per second in large‑scale streaming systems.
This article walks through ten high‑level MySQL tricks—including EXPLAIN analysis, advanced indexing, window functions, CTEs, JSON handling, partitioning, join optimizations, user‑defined variables, online DDL, and generated columns—showing concrete examples, performance trade‑offs, and step‑by‑step guidance for turning slow queries into lightning‑fast operations.
This article explains MySQL partition types (RANGE, LIST, HASH, KEY), demonstrates partition pruning, details maintenance commands, presents real‑world use cases such as time‑series and massive tables, and offers best‑practice recommendations for key selection, partition count, automation, and query optimization.
This comprehensive tutorial walks you through preparing the boot USB, configuring BIOS/UEFI, selecting installation options—including dual‑boot, full‑disk erase, or manual partitioning—setting language, network, and drivers, and completing post‑install updates and driver checks to ensure a smooth Ubuntu 24.04 LTS setup.
This article explains how backend engineers should rethink table design, indexing, partitioning, and key strategies when migrating attendance data from MySQL's OLTP model to Apache Doris 2.1.7's OLAP architecture, providing concrete DDL examples and practical tips.
This article presents eleven practical MySQL advanced techniques—including EXPLAIN analysis, sophisticated indexing, window functions, CTEs, JSON handling, partitioning, join optimizations, user variables, online DDL, and generated columns—to help developers dramatically improve query performance and scalability in high‑load environments.
This article explains how Kafka leverages sequential disk I/O, zero‑copy data transfer, OS page cache, and partition‑based parallelism to deliver massive throughput, detailing the underlying mechanisms, performance numbers, and a practical formula for estimating total system capacity.
The article reveals seven common PostgreSQL architectural oversights—such as neglecting vacuum, misusing UUID primary keys, treating the database as a queue, missing indexes, over‑relying on ORMs, ignoring write‑side scaling, and failing to partition large tables—that silently degrade performance and reliability, and provides concrete fixes and best‑practice configurations.
This guide explains the core data‑warehouse terminology—from ODS, DWD, DWS, and ADS layers, through fact and dimension tables, SCD handling, partitioning and bucketing, to data quality dimensions, common data issues, ETL vs. ELT, data lake vs. warehouse, and OLTP/OLAP concepts—providing a complete reference for new data engineers.
This guide walks senior backend engineers through the strategic reasoning, capacity estimation, and step‑by‑step migration techniques for sharding databases, covering when to split, choosing between partitioning, read/write splitting, or full sharding, and how to plan safe expansions.
This article explains Hive's partitioning and bucketing techniques, compares their purposes, advantages, and pitfalls, and shows how to combine them with concrete SQL examples to improve query performance, reduce I/O, and optimize joins and sampling in large data warehouses.
The article recounts how a PostgreSQL instance on a modest 2‑CPU, 4 GB VM handling 1.2 billion rows was dramatically accelerated by adding BRIN indexes, scheduling maintenance with pg_cron, applying query folding and tuning memory and parallel settings, achieving sub‑second query times without additional hardware.
This guide walks you through why proper Linux partitioning matters, presents production‑grade and scenario‑specific layouts, covers hardware checks, BIOS/UEFI settings, step‑by‑step partitioning with fdisk/GPT and LVM, filesystem choices, security hardening, automation scripts, monitoring, and best‑practice tips for a flawless system install.
This article explains why MySQL partitions based on a timestamp column still function correctly even when the application does not explicitly set the column, by detailing the default CURRENT_TIMESTAMP behavior, partition key evaluation, and practical examples with table definitions and insert statements.
The article explains how Kafka achieves high throughput by writing messages sequentially to disk, leveraging OS page cache and zero‑copy system calls, using partitioned topics for parallelism, batching and compressing records on both producer and broker sides, and employing asynchronous replication with configurable persistence strategies.
This article explains the key techniques Kafka uses to maximize throughput, including sequential disk writes with zero‑copy, partition‑based parallelism, batch processing with compression, and configurable asynchronous replication for balancing performance and reliability.
This article explains the concepts, differences, advantages, and disadvantages of database sharding and partitioning, provides practical MySQL examples for horizontal sharding by region and range partitioning, and offers guidance on choosing the right approach for scalability and maintainability.
This article explains why deleting tens of millions of rows is challenging, analyzes transaction, lock, resource, and business impacts, and presents five practical solutions—including batch deletion, table rebuild, partition dropping, online migration, and professional tools—along with best‑practice guidelines and a decision matrix.
This guide explains MySQL partitioning fundamentals, shows how to create RANGE, LIST, and KEY partitions with concrete CREATE TABLE examples, compares the performance benefits and maintenance trade‑offs, and clarifies the differences between partitioning and sharding to help you decide when to use each technique.
This guide walks you through why careful partition planning is critical, presents production, desktop, development, and cloud server layouts, details hardware checks, BIOS settings, command‑line and LVM partitioning steps, compares filesystems, and offers security hardening, automation scripts, and monitoring tips for a flawless Linux installation.
This article explains why consumer instances cannot scale indefinitely, how to plan partition numbers, fast‑track solutions for message backlog, consumer‑side performance tweaks, and an advanced asynchronous consumption model, providing a complete, interview‑friendly framework for handling MQ congestion.
This article explains how Spring Batch’s core concepts, chunk processing, metadata management, virtual‑thread integration, partitioning, and multi‑threaded step configurations can dramatically increase throughput and resource efficiency when handling large, I/O‑intensive data sets.
This article explains Hive's partitioning mechanisms—including static, dynamic, and multi-level partitions—by showing how to create partitioned tables, load data with explicit or inferred partition values, compare query performance, and follow best practices for effective data organization.
This article explores the fundamentals of message ordering in distributed message queues, explains why ordering is determined by broker arrival, compares global and partial ordering, and presents practical solutions—from single-partition designs to multi-partition hashing, handling data skew, and safe expansion—plus interview tips.
This article details the challenges of migrating 100,000+ SQL statements from BigQuery to MaxCompute for a leading Southeast Asian tech group, explains the new MaxCompute SQL syntax, auto‑partition and ingestion‑time tables, enhanced built‑in functions, and the BigQuery compatibility mode that ensures seamless query behavior.
This guide covers essential Linux storage management techniques, including how to list devices, mount and unmount filesystems, use the find command for file searches, create and manage disk partitions, configure swap space, and set up disk quotas for user limits.
This article demonstrates how to use Spring Batch partitioning in Spring Boot 3, covering the architecture of manager and worker steps, custom partitioner implementation, job and step configuration, essential beans, and a complete runnable example that processes millions of records with parallel threads.
This guide explains how to efficiently produce messages with TDMQ CKafka, covering topic creation, partition sizing, retry policies, asynchronous sending, key/value handling, batch tuning, sticky partitioning, idempotence, ACK settings, code examples, and strategies to avoid data skew and ensure high‑throughput, low‑latency messaging.
Drawing on a decade of operations experience, this guide walks you through zero‑incident Linux system installation and optimal partitioning, covering hardware checks, BIOS settings, multiple partition schemes for servers, desktops, cloud, LVM automation, filesystem choices, security hardening, performance tuning, troubleshooting, and post‑install monitoring.
During massive daily imports of up to one million reconciliation records, MySQL deadlocks can cripple performance; this article dissects the four classic deadlock conditions, then presents a systematic “One‑Order Four‑Split” strategy—ordered writes, transaction splitting, index management, partitioning, and gap‑lock removal—backed by benchmarks, code samples, and configuration tweaks that cut lock rates by up to 90% and reduce batch times to under ten minutes.
This article explores the performance bottlenecks of a 700‑million‑row MySQL orders table and presents a comprehensive sharding solution—including vertical and horizontal partitioning, gene‑based shard keys, routing logic, global secondary indexes, data migration strategies, and common pitfalls—demonstrating how query latency can drop from seconds to milliseconds while supporting billions of records.
This guide explains Linux storage concepts, covering mechanical and solid‑state disks, hot‑plug options, interfaces, and partitioning methods (MBR and GPT), and provides step‑by‑step commands for creating, formatting, and mounting partitions with examples of fstab configuration.
This article breaks down the core techniques that give Kafka its high‑throughput capability, including producer batch settings (batch.size, linger.ms), broker append‑only writes, consumer poll configuration, partition distribution, zero‑copy data transfer, dual‑thread processing, and configurable compression algorithms.
This article details how the activity middle‑platform team identified the root causes of slow SQL queries—such as missing indexes, large data volumes, and inefficient joins—and applied data cleaning, partitioning, indexing, and query refactoring to slash daily slow‑query counts from thousands to just dozens.
The article outlines why deleting tens of millions of rows in one statement harms performance and reliability, then presents safe, efficient tactics—such as indexed batch deletes, partition drops, creating a new table, or using TRUNCATE—plus pre‑deletion planning, post‑delete validation, monitoring, and space reclamation.
This article explains the risks of bulk deletions on million‑row tables and provides a step‑by‑step guide—including pre‑deletion planning, indexing, backup, batch deletion, partitioning, table recreation, TRUNCATE, and post‑deletion verification—to safely and quickly remove large volumes of data while minimizing lock contention, log growth, CPU spikes, replication lag, and rollback complications.
This article examines whether adding an index to a status field in a multi‑million‑row MySQL table improves query performance, covering the index lookup process, extreme cases, selectivity, result‑set size impact, composite indexes, partitioning, and how to verify plans with EXPLAIN.
When a MySQL order table with 20 million rows is paginated using a large OFFSET, query latency can explode from seconds to minutes, but applying six concrete techniques—cursor pagination, deferred join, covering index, table partitioning, precomputed pages, and Elasticsearch integration—can shrink response time to a few hundred milliseconds.
This guide explains Linux disk management fundamentals, covering device files, disk types, RAID levels, partitioning methods, filesystem choices, mounting techniques, useful command‑line tools, real‑world case studies, and step‑by‑step LVM creation and expansion.
This guide walks through Linux storage fundamentals—including device identification, viewing, mounting (temporary and permanent), file searching, disk partitioning methods, swap setup, device removal, and disk quota configuration—providing practical commands and best‑practice tips for system administrators.
This guide explains Linux disk physical and data structures, partitioning reasons and types, key file systems like XFS and swap, essential commands such as fdisk, mkfs, mount, umount, and df, and how to configure automatic mounting with /etc/fstab.
This article distills the key design patterns behind distributed systems—covering replication, partitioning, consensus, and fault‑tolerance—by presenting each pattern’s problem statement, concrete solution, trade‑offs, and technical considerations, all illustrated with real‑world examples from projects like Kafka and Cassandra.
This article explains how adding an index to a status column in a tens‑of‑millions‑row MySQL table affects query performance, covering the basic index lookup process, extreme cases where full scans are chosen, selectivity, covering indexes, composite indexes, partitioning, and using EXPLAIN to verify execution plans.
The article explains how using a large OFFSET with MySQL LIMIT forces the engine to scan and sort hundreds of thousands of index and data pages, then presents three concrete optimizations—covering index scans, subqueries, and partitioned tables—to dramatically reduce the work.
This guide explains PostgreSQL table partitioning, covering its definition, benefits, drawbacks, the three partition types (RANGE, LIST, HASH), step‑by‑step SQL commands for creating partitions, indexes, and managing them, and concludes with practical recommendations for effective use.
This article explains how to assess MySQL table size using table capacity, disk usage, and instance resources, describes why large tables degrade performance, and presents three practical solutions—partitioning, sharding (horizontal/vertical), and hot‑cold data archiving—along with their trade‑offs and implementation tips.
This article explains how Hive serves as a data‑warehouse layer for user‑profile tagging, covering data‑warehouse fundamentals, fact‑and‑dimension modeling, partitioned storage, label aggregation, and ID‑mapping techniques with practical Hive DDL/DML examples.
This article examines the replication and partitioning designs of Redis, Kafka, and Elasticsearch, drawing on concepts from Designing Data‑Intensive Applications to illustrate core distributed‑system principles, common challenges, and practical configuration options.
This guide explains the physical structure of mechanical hard drives, how data is organized into sectors, tracks, and cylinders, and provides step‑by‑step Linux commands for calculating disk capacity, creating MBR/GPT partitions, formatting filesystems, and managing swap space.
This article explains the concepts of database sharding and partitioning, compares common middleware solutions, and offers practical guidance on choosing and implementing horizontal or vertical splits to handle rapid growth and high concurrency in modern applications.
This article examines GitLab’s extensive PostgreSQL schema, explaining why the project uses a pure‑SQL structure file, how primary‑key choices, internal versus external IDs, naming conventions, timestamps, text types, partitioning, GIN indexes, JSONB columns, foreign‑key strategies and other data‑type decisions impact performance, maintainability, and scalability.
This article investigates a slow‑SQL incident caused by the TDDL optimizer reordering a MySQL query, analyzes why the optimizer chose an inefficient index, demonstrates how to verify and force the correct index, and proposes better index and partition‑key designs for optimal performance.
This guide walks you through Linux disk management fundamentals, covering how to partition disks with fdisk and gdisk, create filesystems using mkfs, format partitions, mount and unmount devices, configure automatic mounts via /etc/fstab, and handle special cases such as ISO and Windows share mounts.
This guide explains the physical structure of mechanical hard drives, how data is organized into sectors, tracks, and cylinders, compares old and new zoning methods, and provides detailed Linux commands for calculating capacity, creating partitions with MBR or GPT, formatting filesystems, and managing mounts and swap space.
This article explains the fundamentals of database sharding and partitioning, why they are needed for massive data volumes, various vertical and horizontal splitting strategies, routing algorithms, client vs. proxy deployment models, and the practical challenges and solutions when using ShardingSphere 5.x.
This guide walks you through using StarRocks—covering effortless expression‑based partitioning, streamlined data loading with INSERT FROM FILES and PIPE, powerful in‑flight data transformation using SELECT/JOIN/UNNEST, and flexible table structure tweaks via ALTER TABLE to boost query performance.
This comprehensive guide explains Linux disk management fundamentals, covering how to partition disks with fdisk and gdisk, format them using mkfs, mount and unmount filesystems, configure automatic mounts via /etc/fstab, and handle special cases such as ISO images and Windows shares.
This comprehensive guide explains why MySQL sharding (分库分表) is essential for handling massive data growth, outlines various horizontal and vertical partitioning strategies, discusses common challenges such as hot‑spot, cross‑database queries, pagination, and transaction handling, and compares popular middleware solutions.
This article explains how to assess MySQL table size through table capacity, disk usage, and instance resources, illustrates the impact of B+‑tree index depth on query speed, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—to improve performance.
This article explains Doris’s prefix index mechanism, best practices for partitioning and bucketing, and how to correctly use Flink’s batch writes with sequence columns to ensure ordered updates, providing practical guidance for optimizing OLAP workloads in Apache Doris.
This article explains the concepts of local and global indexes in OceanBase, demonstrates how to create tables and indexes, compares six practical scenarios—including no index, local index, partitioned queries, unique and global indexes—and provides performance metrics to guide database optimization.
This article explains how to design efficient StarRocks tables with proper field types, partitioning and bucketing, compares update and primary‑key models, presents real‑world cases of memory and tablet issues, provides a complete table‑creation example, and outlines comprehensive monitoring metrics to keep the analytical data warehouse performant and stable.
This article examines why MySQL can still handle massive tables and presents practical techniques—including proper primary keys, eliminating redundant indexes, choosing optimal data types, compression, archiving, partitioning, sharding, query optimization, and safe schema changes—to keep large‑table performance high.
This article explains how to optimize OLAP workloads by balancing normalization and denormalization, applying data sharding, replication, indexing, partitioning, materialized views, columnar storage, compression, and lifecycle management, and provides practical ClickHouse SQL examples for index creation, partitioning, and query plan analysis.
This guide outlines a comprehensive data model architecture for data warehouses, detailing layer definitions (ODS, CDM, DWD, DWS, ADS), naming conventions, data type standards, partitioning strategies, redundancy rules, and lifecycle management policies to ensure consistency, performance, and maintainability across big‑data environments.
This article provides a comprehensive overview of Kafka, covering its core architecture, message queue models, communication process, partition selection, consumer groups, rebalancing strategies, partition assignment algorithms, reliability guarantees, replica synchronization, and reasons for removing Zookeeper in newer versions.
This article explains the concept, motivations, design methods, common patterns, and trade‑offs of database sharding (both table and database level), offering practical code examples, scalability criteria, and guidance on avoiding over‑engineering while considering alternatives like TiDB.
This article explains how Apache Doris implements data sharding, partitioning (both list and range, static and dynamic), batch partition creation, and the Auto Bucket feature, providing detailed syntax, algorithms, and performance benefits for large‑scale distributed databases.
This guide walks you through Oracle's storage architecture, explains when and why to use partitioned tables, details each partition type, and provides step‑by‑step SQL examples for creating, managing, and modifying range, list, hash, and composite partitions.
This article presents the background, overall architecture, partitioning strategies, raft‑based replication, binlog support, multi‑active deployment, bulk‑load mechanisms, storage‑engine optimizations, load‑balancing policies, and failure‑detection & recovery techniques of a high‑reliability, high‑throughput key‑value store used at Bilibili.
This article presents a practical guide to implementing monthly sharding for high‑volume billing data using MySQL stored procedures, dynamic table creation, and SpringBoot‑MyBatis integration, covering schema design, code examples, and query strategies for inserting, retrieving, and managing partitioned tables.
This article explains how to classify massive order data into hot and cold segments, store them in MySQL, Elasticsearch and Hive respectively, and implement sharding and partitioning at both table and database levels using modulo and hash calculations to achieve scalable performance for billions of orders.
This article explains the principles, benefits, and challenges of master‑slave, multi‑master, and leaderless database replication, discusses replication lag, quorum consistency, conflict resolution, and presents various data partitioning techniques with practical examples and diagrams.
This article explains why and when to apply database sharding, distinguishes between sharding databases, tables, or both, describes horizontal and vertical splitting, guides key selection and sharding algorithms, covers global ID generation, lists open‑source tools, and discusses the operational challenges introduced by sharding.
The article examines MySQL partition tables that lack primary keys, compares query performance against tables with primary keys, and provides guidance on index design for such partitioned tables, concluding that partition keys need not be included in indexes.
This article details the design and implementation of a high‑availability, high‑performance membership system that serves billions of users across multiple platforms, covering Elasticsearch dual‑center clusters, traffic‑isolated three‑cluster setups, Redis caching strategies, MySQL dual‑center partitioning, and advanced flow‑control and degradation mechanisms.
The article explains how RC and snapshot isolation fail to prevent write skew and phantom reads, advocates serializable isolation, and examines three implementation techniques—strict serial execution, two‑phase locking, and optimistic concurrency—detailing their trade‑offs, stored‑procedure usage, partitioning, and practical constraints for achieving true serializable isolation.
This article explains the four key techniques—page‑cache usage, sequential disk writes, zero‑copy transfers, and partitioned segment indexing—that enable Kafka to achieve exceptionally high write performance, detailing how each mechanism reduces latency and maximizes throughput.
This comprehensive guide covers Kafka’s core concepts, architecture, Zookeeper role, producer sending modes, partitioning strategies, replica types, message deletion, performance optimizations, consumer models, offset management, and best‑practice recommendations for scaling and ensuring ordered delivery in distributed streaming systems.
This article explains Doris's fundamental concepts such as rows, columns, tablets, and partitions, provides guidelines for column definition, partitioning and bucketing strategies, and details table creation syntax and property settings for optimal big‑data storage and query performance.
This article provides a comprehensive English guide to Kafka's producer component, covering its end‑to‑end workflow, detailed steps such as record creation, serialization, partitioning, batching, sender thread, custom data structures, send modes, common exceptions, and essential configuration parameters for optimal performance.
This article walks through preparing an old Windows 7 laptop for a dual‑boot setup, covering disk partitioning, downloading Ubuntu, creating a bootable USB with Rufus, configuring BIOS/UEFI, and completing the Ubuntu installation with detailed screenshots and troubleshooting tips.
This article explains Kafka's core architecture, covering the logical concept of topics, physical partitioning and replication, leader‑follower mechanics, consumer groups, log segmentation with zero‑copy I/O, and how Zookeeper manages broker registration, topic metadata, consumer coordination, and offset tracking, while also discussing producer and consumer load‑balancing strategies.
This article provides a comprehensive overview of Kafka’s core components—including brokers, topics, partitions, producers, and consumers—explaining their roles, log structures, replication mechanisms, and the system’s architecture, supplemented with practical visual illustrative diagrams.
This article walks you through preparing hardware, partitioning the disk, downloading an Ubuntu ISO, creating a bootable USB with UltraISO or Rufus, configuring BIOS/UEFI settings, and completing the Ubuntu installation alongside an existing Windows 7/10 system.
This article explains why large‑scale systems need data partitioning beyond replication, defines partitions and sharding, compares key‑range and hash‑based partitioning methods, discusses load‑balancing and hotspot avoidance, and shows how replication works together with partitioning to improve scalability.
This article introduces Apache Doris, a high‑performance, real‑time analytical database built on an MPP architecture, covering its simple FE/BE design, three data‑model types (Aggregate, Unique, Duplicate), partitioning and bucketing strategies, rollup tables, and the limitations and best practices of row‑level updates.
This article explains MySQL's support for multi‑column partition tables, provides a stored‑procedure to create them, compares performance against single‑column partitions under various query conditions, discusses partition column order impact, and offers practical guidance for when and how to use multi‑column partitions.
