This article explains how to extend Redis beyond a single instance by covering partitioning, master‑slave replication, Sentinel automatic failover, and Redis Cluster, describing their usage methods, advantages, and drawbacks for high‑traffic, high‑availability scenarios.
This article explores multiple MySQL techniques—triggers with a counter table, range partitioning, general tablespace limits, and application‑side buffering—to enforce a hard cap of 10,000 rows per table, discussing implementation steps, code examples, and the trade‑offs of each method.
This article explains how to achieve high‑throughput, high‑availability ordered messaging in RocketMQ by using multi‑partition designs, routing strategies, and consumer locking mechanisms to guarantee local order while handling scaling, failures, and consumer concurrency.
This article provides a comprehensive introduction to Apache Kafka, covering its core concepts, terminology, data flow, configuration options such as compression and partitioning, producer and consumer behavior, replication, consumer groups, offset management, and log retention strategies.
This article provides a comprehensive technical overview of Apache Kafka, covering its distributed messaging architecture, key features such as high‑throughput read/write, replication, partitioning, consumer group mechanics, offset management, rebalance processes, and practical code examples for synchronous and asynchronous offset commits.
This article presents a series of MySQL experiments that examine how different isolation levels, partition‑key usage, and comparison operators affect the number and scope of locks acquired on partitioned tables, revealing the impact of partition pruning on lock contention.
This comprehensive guide covers MySQL single‑table optimization, field and index design, query best practices, engine choices, system parameters, hardware scaling, read/write splitting, caching strategies, partitioning, vertical and horizontal sharding, client vs. proxy architectures, and alternative databases, providing actionable steps for high‑performance data handling.
This article compares Redis list, Kafka, and Pulsar as messaging middleware, explaining their architectures, strengths, and weaknesses—including queue fundamentals, partitioning, cursor management, consumer groups, high‑availability mechanisms, storage strategies, and consumption models—to help readers choose the right solution for large‑scale systems.
This article explains the fundamentals of distributed systems, covering why they arise, vertical and horizontal scaling, system splitting, sharding and partitioning strategies, load balancing, replication, the CAP theorem, and the BASE model, all illustrated with practical examples and code snippets.
This article describes how to improve the performance of a massive MySQL 5.6 user‑log table by redesigning schema, applying proper indexes, using partitioning, considering table sharding, and evaluating upgrade paths to compatible cloud or big‑data databases, with concrete SQL examples and cost analysis.
This case study explains how a client with a 2‑billion‑row MySQL table avoided disk exhaustion and poor query performance by streaming data through Kafka to Google BigQuery, partitioning the data, and cleaning it to reclaim storage—all without downtime.
Apache Kafka is a distributed, fault‑tolerant streaming platform that uses broker clusters, topic partitions, and replicated logs to provide publish/subscribe and queue messaging, configurable retention, strong ordering within partitions, producer acknowledgments, consumer groups, and performance optimizations such as batching and zero‑copy for real‑time data pipelines.
Facing a MySQL table with over 2 billion continuously growing rows, the team designed a cloud‑based solution using Kafka to stream data into Google BigQuery, applied partitioned schemas and data cleaning to improve query performance, reduce storage costs, and avoid downtime.
This guide walks through the complete process of adding a new disk to a Linux server running on VMware, including disk addition, partitioning, formatting, mounting, and handling disks larger than 2 TB with detailed command examples.
When a MySQL table grows large, performance degrades sharply, so this guide explains single‑table tuning, proper field choices, index strategies, query best practices, engine differences, system parameters, hardware upgrades, read‑write splitting, caching layers, table partitioning, vertical and horizontal sharding, and how to choose suitable sharding solutions.
This article provides a comprehensive guide to improving MySQL performance by optimizing single‑table schemas, indexes, queries, engine settings, system parameters, and then discusses read/write separation, caching layers, table partitioning, vertical and horizontal sharding, compatible scalable databases and when to consider NoSQL alternatives.
This article examines the challenges of performing time‑series similarity queries on massive datasets and presents three scalable solutions—partition‑based indexing, dimensionality‑reduction using MinHash, and a combined approach with Locality Sensitive Hashing—to reduce computation while preserving similarity accuracy.
This article details how Vivo's cloud service tackled explosive data growth by applying horizontal and vertical sharding, routing‑table based dynamic expansion, and MySQL InnoDB compression, providing step‑by‑step guidance, performance results, and practical recommendations for large‑scale database deployments.
This article details how to improve performance of massive MySQL tables on Alibaba Cloud by redesigning schemas and indexes, migrating to PolarDB with read‑write separation, partitioning historical data using the X‑Engine storage engine, and leveraging PolarDB's parallel query capabilities for faster analytics.
When a MySQL table grows large, CRUD performance degrades, so this article presents comprehensive optimization techniques—including field design, indexing, query rewriting, engine selection, system parameters, hardware upgrades, read‑write separation, caching, partitioning, vertical and horizontal splitting, sharding architectures, and recommendations for compatible scalable databases—to help maintain high performance and scalability.
This article explains how modern in‑memory database systems structure data into fixed‑ and variable‑length blocks, choose partitioned or non‑partitioned architectures, handle row versus column storage, and implement cache‑aware index structures such as CSB+-Tree, PB+-Tree, Bw‑Tree, Adaptive Radix, OLFIT and Skiplists, illustrated with Hekaton, H‑Store/VoltDB, HyPer and SAP HANA examples.
This article traces the evolution of openGauss from its Huawei roots and systematically evaluates how well openGauss 1.0.0 supports Oracle’s various table types, providing creation examples, compatibility results, and guidance on work‑arounds for unsupported features.
This article analyzes common database performance bottlenecks such as I/O and CPU limits, explains horizontal and vertical sharding techniques, reviews available sharding tools, discusses the challenges of distributed transactions, cross‑node joins, pagination, and global primary‑key generation, and offers practical mitigation strategies.
This article presents three HiveSQL optimization case studies—restructuring a large‑scale query with partitioned tables, breaking a complex window‑function query into smaller subsets with joins, and refactoring excessive PERCENTILE_APPROX calls—demonstrating how each change reduces execution time from hours to minutes and improves overall performance.
This article analyzes a severe MySQL performance issue caused by massive user‑access logs, then presents three solution paths—optimizing the existing database, migrating to a compatible high‑performance database, and adopting big‑data technologies—detailing design best practices, indexing, partitioning, sharding, and cloud‑native options to restore query speed and scalability.
This article analyzes the performance problems of a massive MySQL 5.6 RDS table storing billions of user‑activity records and presents three practical solutions—optimizing the existing schema and queries, migrating to a MySQL‑compatible high‑performance database, and adopting a big‑data platform—detailing design, indexing, partitioning, sharding, and cloud‑native options.
This article explains the InnoDB logical storage hierarchy of segments, extents, and pages, then details MySQL horizontal partitioning concepts, types (RANGE, LIST, HASH, KEY), example DDL and DML, and discusses how partitioning impacts performance for OLTP and OLAP workloads.
This article examines performance problems with a 20‑million‑row MySQL user‑log table on Alibaba Cloud RDS, outlines three solution paths—optimizing the existing database, migrating to a MySQL‑compatible high‑performance service, and adopting a big‑data engine—and provides detailed guidance on schema design, indexing, partitioning, and practical SQL tweaks.
This article explains Redis sharding, covering its purpose, various partitioning methods such as range and hash partitioning, implementation approaches like client‑side and proxy‑assisted sharding, their drawbacks, and practical steps for migration and using Redis Cluster or Twemproxy.
This article examines why a MySQL 5.6 RDS table with tens of millions of rows becomes unbearably slow, then presents three concrete approaches—optimizing the existing database, migrating to a MySQL‑compatible service, and adopting a big‑data engine—detailing design, indexing, partitioning, sharding, and cloud options to restore performance.
This article explains how to extract maximum performance and scalability from PostgreSQL for multi‑terabyte workloads by leveraging specialized indexes, declarative partitioning, tablespaces, parallel query execution, read‑only replica load‑balancing, and foreign‑table sharding techniques.
This article explains how ClickHouse's MergeTree engine creates partition directories, the naming convention for partitions, and the automatic and manual merging processes that consolidate parts while tracking block numbers and merge levels.
This article explains common I/O and CPU bottlenecks in databases, introduces horizontal and vertical sharding concepts, compares sharding tools, outlines practical sharding steps, discusses typical sharding issues such as non‑partition queries and expansion, and provides a concise summary and example implementation.
This document outlines comprehensive MySQL database design standards and SQL writing guidelines, covering naming conventions, character sets, table and column specifications, index design, partitioning, data types, transaction handling, and prohibited statements to ensure stable, performant, and maintainable production databases.
The article explains how Kafka achieves high throughput by using partition‑level parallelism, sequential disk writes with segment files, extensive use of the OS page cache, zero‑copy data paths, request batching and optional compression, while also discussing the underlying disk I/O principles.
This article provides an extensive overview of Apache Kafka, covering its use cases, key concepts such as ISR, AR, HW, LEO, and LW, message ordering, the roles of partitioners, serializers and interceptors, producer and consumer client architecture, offset handling, multithreaded consumption, and topic partition management.
This article provides a thorough overview of MySQL performance tuning, covering proper table design according to the three normal forms, various index types and their creation, partitioning strategies, read/write separation, storage engine selection, and practical SQL tips for improving query efficiency.
This guide explains how to add virtual disks to a Linux VM, identify them, create partitions with fdisk, format them with XFS or ext4, mount and unmount the filesystems, and configure automatic mounting at boot using /etc/fstab.
This article presents comprehensive techniques for improving MySQL performance on large single tables, covering column choices, index design, query writing, engine selection, system parameters, hardware upgrades, read‑write splitting, caching, table partitioning, vertical and horizontal sharding, and guidance on selecting mature sharding solutions.
This article presents a thorough guide to improving MySQL performance for large single tables, covering field choices, indexing best practices, query tuning, engine selection, system parameters, hardware upgrades, read/write separation, caching layers, partitioning, vertical and horizontal sharding, and compatible scalable database alternatives.
ByteKV is a C++-based, strongly consistent, range-partitioned key‑value storage system built by ByteDance, featuring a multi‑Raft consensus layer, custom storage engines (RocksDB and BlockDB), automatic partition splitting/merging, load balancing, distributed transactions, and a SQL table layer for rich data models.
This article explains how to extend Redis beyond single‑node limits by using partitioning, master‑slave replication, Sentinel for automatic failover, and Redis Cluster with hash slots, detailing their usage, advantages, drawbacks, and configuration examples for building high‑availability and scalable in‑memory data stores.
This article explains database bottlenecks caused by IO and CPU limits, introduces horizontal and vertical sharding for databases and tables, compares popular sharding tools, discusses challenges such as distributed transactions, cross‑node joins, pagination and global ID generation, and offers guidance on when and how to apply sharding in real‑world systems.
This guide walks through essential Oracle Database administration tasks, covering how to start and shut down the database and listener, create and manage tablespaces, users, and permissions, control transactions, build and maintain indexes, views, materialized views, sequences, synonyms, and partitioned tables, with command examples and best‑practice tips.
This article explains InnoDB's logical storage hierarchy of tablespaces, segments, extents, and pages, then details MySQL's partitioning concepts, types, creation syntax, and performance considerations, providing practical examples and guidance for effective database design.
This article provides a comprehensive overview of Kafka's Producer side, covering its architecture, the ProducerRecord data structure, serialization mechanisms, partitioning logic, and the accumulator buffer, while comparing old and new Producer clients and illustrating key configurations with code examples.
When a MySQL table grows to millions of rows, query, insert, update and delete performance degrades sharply, but by applying proper field design, index strategies, SQL best practices, engine selection, system parameters, caching, partitioning, vertical and horizontal sharding, and choosing suitable client‑side or proxy‑side architectures, the database can efficiently handle tens of millions of records.
This article explains InnoDB's logical storage hierarchy—tablespace, segment, extent, page—then details MySQL partitioning concepts, types, example DDL, performance impacts, and best‑practice tips for OLTP and OLAP workloads.
This guide walks you through creating a bootable Ubuntu USB, allocating disk space, installing Ubuntu alongside Windows, adjusting the boot order, and safely removing the Linux partitions and boot entries to restore a single‑OS Windows system.
This extensive tutorial walks through Kafka producer and consumer fundamentals, demonstrates how to configure key‑based and custom partitioning, explains offset management, consumer group coordination, and essential configuration parameters, and includes complete Java code examples for real‑world e‑commerce scenarios.
Tencent Cloud CKafka powers Tencent Classroom’s pandemic‑era online teaching by replacing a custom queue with a high‑performance, highly available, partition‑based message bus that scales to millions of real‑time interactions, offers configurable replication and tuning for reliability, and integrates with big‑data and streaming tools for analytics.
This article explains the InnoDB logical storage architecture—including segments, extents, and pages—then introduces MySQL table partitioning concepts, details the main partition types (RANGE, LIST, HASH, KEY) with SQL examples, and discusses their performance implications for OLTP and OLAP workloads.
Flink’s StreamPartitioner abstracts data routing in DataStream, offering eight built‑in partitioners—including Global, Shuffle, Rebalance, KeyGroup, Broadcast, Rescale, Forward, and Custom—each with distinct channel selection logic, illustrated with source code snippets and explanations of their runtime behavior.
Data skew in Spark and Hadoop occurs when a few keys dominate shuffle traffic, causing slow tasks, OOM errors, and job failures; the article describes how to detect skew via UI metrics or sampling and offers mitigation tactics such as filtering keys, increasing shuffle partitions, custom partitioners, broadcast joins, salted keys, and Hadoop‑specific settings.
This article explains how IO and CPU bottlenecks increase active database connections, describes horizontal and vertical sharding techniques, outlines practical steps and tools for implementing sharding, and discusses common challenges such as non‑partition key queries and expansion strategies.
This article explains how to build, deploy, and continuously update event streaming pipelines using Spring Cloud Data Flow and Apache Kafka, covering common topologies, named destinations, parallel and partitioned streams, function composition, multiple input/output bindings, and practical shell commands for registration and management.
PostgreSQL 12 brings major enhancements—including up to 40% faster queries on large datasets, improved indexing, partitioning, JSON path queries, generated columns, pluggable table access methods, stronger authentication, and built‑in JIT compilation—while also adding better localization, multi‑constraint authorization, and convenient maintenance commands.
After initially implementing table sharding, we later added database sharding, sharing a detailed retrospective that covers when to partition tables, various sharding strategies (range, hash, range+hash), migration challenges, business compatibility, and final archiving practices, illustrated with diagrams and practical tips.
This article provides a comprehensive guide to MySQL performance optimization, covering database design principles, field type selection, indexing strategies, storage engine choices, query caching, partitioning methods, master‑slave replication setup, read/write splitting, load balancing, and profiling tools for diagnosing slow queries.
This guide walks through shrinking the /home partition, extending the root filesystem on a CentOS 7 server, handling busy mount issues, installing missing tools like fuser and xfsdump, resizing XFS, and safely remounting the partitions.
This article explains DBLE's hash‑based sharding algorithm, detailing how logical shards are derived via modulo, mapped to physical shards through a rule.xml configuration and mapFile, compares it with MyCat, and provides development, operational, and configuration best practices along with code examples.
This article explores practical techniques for handling massive, high‑concurrency data workloads, covering relational database limits, read/write separation, vertical and horizontal sharding, key selection, archival to NoSQL stores, and the use of heterogeneous index tables to maintain performance.
This article introduces Apache Kafka as a distributed streaming platform, explaining its three core capabilities, key concepts such as producers, topics, brokers, partitions and consumers, and detailing how messages are delivered, stored in partitions, and consumed by consumer groups.
This guide explains how to construct a client library for Youzan's customized NSQ, covering workflow configuration, nsqd discovery via nsqlookupd, connection establishment, message publishing and consumption, ordered consumption, and the implementation of message tracing with concrete command syntax and response handling.
This article demonstrates how MySQL partition tables dramatically improve query speed for large datasets, compares partition-key, secondary-index, and full‑scan queries, outlines the practical performance gains, and details the numerous limitations and trade‑offs that influence when to adopt partitioning versus sharding.
To address the long processing time caused by uneven Spark partitions when reading Kafka via the Direct approach, this article explains the SPARK‑22056 solution that modifies KafkaRDD.getPartitions to support a configurable 'topic.partition.subconcurrency' parameter, discusses its trade‑offs, and presents alternative repartition and multithreading techniques.
This article explains how OceanBase 2.0’s partitioning architecture, partition‑group feature, and a series of performance optimizations enabled Ant Group’s payment core to handle millions of transactions per second during the Double‑11 shopping festival, while also describing the system’s advantages, results, and future capabilities.
This article examines various approaches to storing massive data, including table partitioning, NoSQL/NewSQL, and database sharding, analyzes their advantages and drawbacks, and presents practical sharding designs for user, forum, and order databases, with considerations for caching, redundancy, and query routing.
A large‑scale delete operation in a MySQL PXC cluster caused transaction rollback, table locks, and flow control, leading to service slowdown, and was resolved by redesigning the cleanup mechanism with partitioned tables, stored procedures, and scheduled events.
This article explains why auto‑increment primary keys are preferred, compares B+‑tree and hash indexes, discusses MySQL partitioning, isolation levels, MVCC, and provides practical tips for designing efficient database tables and indexes.
This article explores MySQL’s advanced capabilities—including partition tables, views, stored procedures, triggers, foreign‑key constraints, bind variables, user‑defined functions, and character‑set considerations—explaining their principles, usage patterns, performance impacts, and best‑practice recommendations for handling large‑scale data.
This article explains why auto‑increment primary keys are preferred in MySQL, how B+Tree and hash indexes differ, the benefits of composite and partitioned indexes, isolation levels, MVCC, row‑level locking, triggers, stored procedures, optimization tips, MyISAM vs InnoDB, table design guidelines, and also covers Redis architecture, persistence, replication, and eviction policies.
This article presents a comprehensive overview of Oracle partitioning, covering basic concepts, the evolution of partition features across versions, practical best‑practice scenarios, and the latest enhancements in Oracle 12c, 12.2, and 18c that improve performance, manageability, and availability.
This article reviews common MySQL pitfalls—including engine choice, table partitioning, index design, slow‑query causes and optimizations, diagnostic tools, and how to recover from accidental data loss—offering practical guidance for developers to improve performance and reliability.
This article explains how to choose MySQL storage engines, apply effective table partitioning and indexing, identify common causes of slow queries, adopt practical SQL optimization tips, use diagnostic tools, and handle data loss or bugs with proper recovery procedures.
This article walks through a real‑world monitoring project that must store over 400 million rows per day in SQL Server 2012, detailing the performance bottlenecks, bulk‑copy tuning, schema redesign, partitioning strategies, index removal and creation, and query‑optimisation techniques that finally achieve sub‑second query times and meet strict latency requirements.
Database sharding, or splitting databases into multiple instances and tables, is essential for scaling backend systems; this article explains its importance, historical context, core concepts, storage stretching techniques, index and consistency handling, and provides a practical design example for transaction systems.
This article explores MySQL’s advanced features—including partition tables, views, stored procedures, triggers, foreign key constraints, bind variables, user‑defined functions, and character set considerations—explaining their principles, usage patterns, performance implications, and practical tips for large‑scale data scenarios.
This guide provides a comprehensive collection of Oracle SQL commands covering connection, query execution, variable handling, formatting, tablespace creation, modification, and external table operations, presented step‑by‑step for database administrators and developers.
This article explains how to handle a 200‑million‑row MySQL 5.6 table by adding indexes, cleaning 99% of old data, converting it to a partitioned table, and using online DDL and auxiliary tables to maintain continuous business operations.
Learn how to boost MySQL performance for massive tables by optimizing schema design, indexing, query patterns, engine selection, system parameters, hardware upgrades, read/write splitting, caching layers, partitioning, vertical and horizontal sharding, and choosing appropriate client or proxy sharding solutions.
A colleague handed over a seemingly simple SQL Server query that ran in five seconds for early dates but never returned for later dates, and the article walks through the diagnosis, execution‑plan analysis, statistics update, and final fix that restored performance.
Effective ETL optimization is crucial for data warehouse construction, and this guide outlines three core strategies—ensuring proper resource configuration, leveraging data characteristics for optimal partitioning and bucketing, and monitoring task execution—providing practical principles, pitfalls, and case studies to maximize ETL efficiency.
This guide explains how to transform an ordinary Oracle table larger than 2 GB into a partitioned table using the online redefinition method (DBMS_REDEFINITION), detailing each step from feasibility checks to final cleanup with example PL/SQL code.
This guide provides a comprehensive, step‑by‑step tutorial on using Oracle SQL*Plus for connecting to databases, querying tables, saving and executing scripts, handling variables, formatting output, managing tablespaces, creating external tables, and defining various partitioning strategies.
This article outlines practical techniques for optimizing large MySQL tables—including column and index tuning, query improvements, engine selection, system parameters, hardware upgrades, caching, partitioning, vertical and horizontal sharding, and discusses client‑side and proxy‑based sharding solutions and compatible databases.
A detailed case study describes how a rapidly growing edit‑log feature caused query times to soar to 30 seconds, and walks through the step‑by‑step investigation, identification of a custom function bottleneck, data‑volume analysis, and the eventual implementation of partitioning, mandatory time filters, and composite indexing to restore acceptable performance.
This guide explains how to verify and configure Oracle's automatic optimizer statistics collection, customize collection windows, tailor statistics for large and partitioned tables, detect stale statistics, and decide when to gather histograms, providing practical SQL and PL/SQL examples.
The article dissects a moderately complex Oracle SQL query from a data‑warehouse, shows how changing a parallel hint forces a full table scan that halves execution time, and then explores partitioning strategies and broader lessons on OLAP performance tuning.
The article explains when MySQL sharding is needed, outlines five practical principles for deciding to split databases or tables, provides real‑world examples, and shares operational tips for implementing horizontal partitioning to improve performance, availability, and manageability.
This article explains how Redis Cluster automatically distributes data across nodes, how clients locate the correct node using slot hashing and hash tags, and how resharding and slot migration are performed without downtime, providing a detailed overview of the partitioning mechanism.
An in‑depth case study of a sluggish SSRS report for Siemens China's MES system reveals how missing indexes, full‑table scans on billion‑row tables, and misguided partitioning cripple performance, and demonstrates practical SQL refactoring, stored‑procedure redesign, and indexing strategies that cut query time from hours to seconds.
This article analyzes a poorly performing MES reporting SQL query that scans billions of rows, identifies indexing and partitioning shortcomings, and proposes a redesign using stored procedures, temporary tables, and query‑writing best practices to achieve sub‑second response times.
This article explains the concept, benefits, methods, and practical considerations of partitioning data across multiple Redis instances, covering range and hash partitioning, consistent hashing, implementation options, drawbacks, and recommended tools such as Redis Cluster and Twemproxy.
This article outlines practical strategies for Oracle DBAs to design, automate, and fine‑tune statistics‑gathering processes in large, high‑concurrency environments, covering automatic options, handling skewed data, partitioned tables, bind variables, and ongoing optimization cycles.
This article examines how differing key‑hash algorithms in MySQL 5.1 versus 5.5/5.6 cause data loss in partitioned tables, describes compatibility fixes with ALTER TABLE … ALGORITHM=1, analyzes a MySQL client crash caused by mis‑redirected dump output, explores GTID_PURGED replication issues and replicate‑do‑db filtering gaps, and evaluates the behavior of OPTIMIZE TABLE on TokuDB tables.
This article presents a detailed case study of an Oracle RAC system performance bottleneck, walks through comprehensive monitoring, storage and partitioning analysis, and outlines a step‑by‑step optimization plan—including KEEP pool tuning and partition redesign—that dramatically improved I/O latency and overall throughput.
This article examines why MySQL 5.5/5.6 fails to retrieve non‑null partitioned rows after a physical upgrade, how the ALTER TABLE … PARTITION BY KEY ALGORITHM=1 fix works, the cause of MySQL client crashes when dump output mixes warnings, GTID_PURGED replication bugs, replicate‑filter side effects, and the behavior of TokuDB’s OPTIMIZE TABLE.
This article presents 58.com’s experience with MySQL under massive data volumes, covering core concepts such as single‑instance, sharding, replication and grouping, common availability and read‑write challenges, detailed sharding implementations for user, post, friend and order tables, and post‑sharding business practices including IN queries, non‑partition key queries, and cross‑database pagination.
This article compiles practical discussions on database sharding and partitioning, covering criteria such as user ID, time, and hash, redundancy techniques, caching with Redis, query handling, pagination, consistency, and distributed transaction approaches for scalable backend systems.
