This article walks you through preparing hardware, partitioning the disk, downloading an Ubuntu ISO, creating a bootable USB with UltraISO or Rufus, configuring BIOS/UEFI settings, and completing the Ubuntu installation alongside an existing Windows 7/10 system.
This article explains why large‑scale systems need data partitioning beyond replication, defines partitions and sharding, compares key‑range and hash‑based partitioning methods, discusses load‑balancing and hotspot avoidance, and shows how replication works together with partitioning to improve scalability.
This article introduces Apache Doris, a high‑performance, real‑time analytical database built on an MPP architecture, covering its simple FE/BE design, three data‑model types (Aggregate, Unique, Duplicate), partitioning and bucketing strategies, rollup tables, and the limitations and best practices of row‑level updates.
This article explains MySQL's support for multi‑column partition tables, provides a stored‑procedure to create them, compares performance against single‑column partitions under various query conditions, discusses partition column order impact, and offers practical guidance for when and how to use multi‑column partitions.
This article explains the reasons for database sharding, compares vertical (scale‑up) and horizontal (scale‑out) partitioning methods, illustrates vertical database/table splits and horizontal partition strategies such as HASH, RANGE, LIST, KEY and composite, and discusses the challenges like distributed transactions, cross‑database joins and increased management overhead.
This article provides a comprehensive guide to Apache Doris, covering how to create databases and tables with single and composite partitions, import data via broker and routine loads, understand its aggregate, uniq, and duplicate data models, and leverage rollup and prefix index features for optimized querying.
This article explains the concepts, motivations, strategies, and challenges of database and table sharding, covering vertical and horizontal partitioning, range and hash algorithms, when to adopt them, and popular middleware solutions for scalable MySQL architectures.
This article provides a comprehensive overview of Kafka, covering its role as a message middleware, core concepts, architecture, replication management, partition handling, producer sending modes, partition assignment strategies, load balancing, reliability mechanisms, consumer models, controller election, and factors that affect its high throughput and potential message loss scenarios.
This article details the background, architecture, core features, and operational practices of a custom high‑reliability, high‑throughput key‑value storage system that combines Raft replication, flexible partitioning, binlog support, bulk loading, and multi‑active deployment for B‑Station's diverse data workloads.
This article provides a comprehensive guide to ClickHouse SQL, covering database creation, table definitions, column defaults, temporary and partitioned tables, view types, DDL commands, data import/export formats, and mutation‑based update and delete operations with practical code examples.
This guide shows how to combine PostgreSQL declarative time‑range partitioning with Citus distributed sharding to create a scalable time‑series database, covering table creation, distribution, automatic monthly partition generation, columnar archiving, maintenance with pg_cron, and performance benefits such as faster inserts, queries, and data expiration.
This article presents a detailed case study of optimizing a massive MySQL table using time‑based partitioning, describing the problem background, partition design, migration scripts, performance benchmarks, and ongoing maintenance procedures for a ten‑year data archive.
Bilibili’s high‑performance distributed KV store combines hash and range partitioning, Raft‑based multi‑replica consistency, and a Metaserver‑managed topology of pools, zones, nodes, tables, shards and replicas, offering features such as partition splitting, binlog streaming, multi‑active replication, bulk loading, KV‑storage separation, and automated load, leader and health balancing for reliable, scalable data services.
This article explains database sharding fundamentals, including terminology, when to adopt sharding, various global ID generation methods such as auto‑increment, UUID, COMB, and Snowflake, different partitioning strategies, challenges like distributed transactions, and practical expansion and migration solutions.
This article reviews MySQL horizontal partitioning, explains five partitioning strategies (HASH, RANGE, KEY, LIST, COMPOSITE), demonstrates how to create and test partitioned tables, compares performance with and without indexes, and covers partition management operations such as adding, dropping, and rebuilding partitions.
This article provides an in‑depth technical guide to Apache Kafka, covering its distributed streaming architecture, core concepts such as topics, partitions, brokers, producers and consumers, reliability guarantees, storage mechanisms, configuration parameters, and consumer assignment strategies, supplemented with Java code examples.
This article explains how to design Hive data warehouse schemas, covering fact and dimension table modeling, partitioned storage strategies, tag aggregation techniques, and ID‑mapping implementations using Hive SQL and UDFs to support user profiling and analytics.
This article explains how MySQL InnoDB partition tables can dramatically boost query speed, simplify routine operations, enable parallel writes, and provide efficient data exchange and cleanup compared to non‑partitioned tables, illustrated with concrete SQL examples and performance metrics.
This article reviews the challenges of building a real‑time data warehouse, compares Lambda and Kappa architectures, introduces Apache Kudu’s master‑tablet design, storage model and partition strategies, and shares practical experiences and future directions for a Kudu‑based streaming analytics platform.
This article explores Linux file system fundamentals, covering physical disk layout, partition strategies, how the OS reduces random‑read latency, the real space cost of empty files and directories, inode usage, block allocation, and practical tips for improving disk I/O performance.
This article introduces various Redis scaling solutions—including basic partitioning, master‑slave replication, Sentinel high‑availability, and Redis Cluster—explaining their concepts, typical usage patterns, configuration commands, advantages, and drawbacks to help developers choose the right approach for high‑traffic environments.
This article explains the main Redis scaling approaches—including simple partitioning, master‑slave replication, Sentinel high‑availability, and native Cluster—detailing their concepts, usage methods, advantages, drawbacks, and providing practical configuration commands and code examples.
This article introduces the main Redis scaling solutions—including simple partitioning, master‑slave replication, Sentinel high‑availability, and Redis Cluster—explaining their concepts, usage patterns, advantages, and drawbacks to help developers choose the appropriate architecture for high‑traffic environments.
This article explains how to extend Redis beyond a single instance by covering partitioning, master‑slave replication, Sentinel automatic failover, and Redis Cluster, describing their usage methods, advantages, and drawbacks for high‑traffic, high‑availability scenarios.
This article explores multiple MySQL techniques—triggers with a counter table, range partitioning, general tablespace limits, and application‑side buffering—to enforce a hard cap of 10,000 rows per table, discussing implementation steps, code examples, and the trade‑offs of each method.
This article explains how to achieve high‑throughput, high‑availability ordered messaging in RocketMQ by using multi‑partition designs, routing strategies, and consumer locking mechanisms to guarantee local order while handling scaling, failures, and consumer concurrency.
This article provides a comprehensive introduction to Apache Kafka, covering its core concepts, terminology, data flow, configuration options such as compression and partitioning, producer and consumer behavior, replication, consumer groups, offset management, and log retention strategies.
This article provides a comprehensive technical overview of Apache Kafka, covering its distributed messaging architecture, key features such as high‑throughput read/write, replication, partitioning, consumer group mechanics, offset management, rebalance processes, and practical code examples for synchronous and asynchronous offset commits.
This article presents a series of MySQL experiments that examine how different isolation levels, partition‑key usage, and comparison operators affect the number and scope of locks acquired on partitioned tables, revealing the impact of partition pruning on lock contention.
This comprehensive guide covers MySQL single‑table optimization, field and index design, query best practices, engine choices, system parameters, hardware scaling, read/write splitting, caching strategies, partitioning, vertical and horizontal sharding, client vs. proxy architectures, and alternative databases, providing actionable steps for high‑performance data handling.
This article compares Redis list, Kafka, and Pulsar as messaging middleware, explaining their architectures, strengths, and weaknesses—including queue fundamentals, partitioning, cursor management, consumer groups, high‑availability mechanisms, storage strategies, and consumption models—to help readers choose the right solution for large‑scale systems.
This article explains the fundamentals of distributed systems, covering why they arise, vertical and horizontal scaling, system splitting, sharding and partitioning strategies, load balancing, replication, the CAP theorem, and the BASE model, all illustrated with practical examples and code snippets.
This article describes how to improve the performance of a massive MySQL 5.6 user‑log table by redesigning schema, applying proper indexes, using partitioning, considering table sharding, and evaluating upgrade paths to compatible cloud or big‑data databases, with concrete SQL examples and cost analysis.
This case study explains how a client with a 2‑billion‑row MySQL table avoided disk exhaustion and poor query performance by streaming data through Kafka to Google BigQuery, partitioning the data, and cleaning it to reclaim storage—all without downtime.
Apache Kafka is a distributed, fault‑tolerant streaming platform that uses broker clusters, topic partitions, and replicated logs to provide publish/subscribe and queue messaging, configurable retention, strong ordering within partitions, producer acknowledgments, consumer groups, and performance optimizations such as batching and zero‑copy for real‑time data pipelines.
Facing a MySQL table with over 2 billion continuously growing rows, the team designed a cloud‑based solution using Kafka to stream data into Google BigQuery, applied partitioned schemas and data cleaning to improve query performance, reduce storage costs, and avoid downtime.
This guide walks through the complete process of adding a new disk to a Linux server running on VMware, including disk addition, partitioning, formatting, mounting, and handling disks larger than 2 TB with detailed command examples.
When a MySQL table grows large, performance degrades sharply, so this guide explains single‑table tuning, proper field choices, index strategies, query best practices, engine differences, system parameters, hardware upgrades, read‑write splitting, caching layers, table partitioning, vertical and horizontal sharding, and how to choose suitable sharding solutions.
This article provides a comprehensive guide to improving MySQL performance by optimizing single‑table schemas, indexes, queries, engine settings, system parameters, and then discusses read/write separation, caching layers, table partitioning, vertical and horizontal sharding, compatible scalable databases and when to consider NoSQL alternatives.
This article examines the challenges of performing time‑series similarity queries on massive datasets and presents three scalable solutions—partition‑based indexing, dimensionality‑reduction using MinHash, and a combined approach with Locality Sensitive Hashing—to reduce computation while preserving similarity accuracy.
This article details how Vivo's cloud service tackled explosive data growth by applying horizontal and vertical sharding, routing‑table based dynamic expansion, and MySQL InnoDB compression, providing step‑by‑step guidance, performance results, and practical recommendations for large‑scale database deployments.
This article details how to improve performance of massive MySQL tables on Alibaba Cloud by redesigning schemas and indexes, migrating to PolarDB with read‑write separation, partitioning historical data using the X‑Engine storage engine, and leveraging PolarDB's parallel query capabilities for faster analytics.
When a MySQL table grows large, CRUD performance degrades, so this article presents comprehensive optimization techniques—including field design, indexing, query rewriting, engine selection, system parameters, hardware upgrades, read‑write separation, caching, partitioning, vertical and horizontal splitting, sharding architectures, and recommendations for compatible scalable databases—to help maintain high performance and scalability.
This article explains how modern in‑memory database systems structure data into fixed‑ and variable‑length blocks, choose partitioned or non‑partitioned architectures, handle row versus column storage, and implement cache‑aware index structures such as CSB+-Tree, PB+-Tree, Bw‑Tree, Adaptive Radix, OLFIT and Skiplists, illustrated with Hekaton, H‑Store/VoltDB, HyPer and SAP HANA examples.
This article traces the evolution of openGauss from its Huawei roots and systematically evaluates how well openGauss 1.0.0 supports Oracle’s various table types, providing creation examples, compatibility results, and guidance on work‑arounds for unsupported features.
This article analyzes common database performance bottlenecks such as I/O and CPU limits, explains horizontal and vertical sharding techniques, reviews available sharding tools, discusses the challenges of distributed transactions, cross‑node joins, pagination, and global primary‑key generation, and offers practical mitigation strategies.
This article presents three HiveSQL optimization case studies—restructuring a large‑scale query with partitioned tables, breaking a complex window‑function query into smaller subsets with joins, and refactoring excessive PERCENTILE_APPROX calls—demonstrating how each change reduces execution time from hours to minutes and improves overall performance.
This article analyzes a severe MySQL performance issue caused by massive user‑access logs, then presents three solution paths—optimizing the existing database, migrating to a compatible high‑performance database, and adopting big‑data technologies—detailing design best practices, indexing, partitioning, sharding, and cloud‑native options to restore query speed and scalability.
This article analyzes the performance problems of a massive MySQL 5.6 RDS table storing billions of user‑activity records and presents three practical solutions—optimizing the existing schema and queries, migrating to a MySQL‑compatible high‑performance database, and adopting a big‑data platform—detailing design, indexing, partitioning, sharding, and cloud‑native options.
This article explains the InnoDB logical storage hierarchy of segments, extents, and pages, then details MySQL horizontal partitioning concepts, types (RANGE, LIST, HASH, KEY), example DDL and DML, and discusses how partitioning impacts performance for OLTP and OLAP workloads.
This article examines performance problems with a 20‑million‑row MySQL user‑log table on Alibaba Cloud RDS, outlines three solution paths—optimizing the existing database, migrating to a MySQL‑compatible high‑performance service, and adopting a big‑data engine—and provides detailed guidance on schema design, indexing, partitioning, and practical SQL tweaks.
This article explains Redis sharding, covering its purpose, various partitioning methods such as range and hash partitioning, implementation approaches like client‑side and proxy‑assisted sharding, their drawbacks, and practical steps for migration and using Redis Cluster or Twemproxy.
This article examines why a MySQL 5.6 RDS table with tens of millions of rows becomes unbearably slow, then presents three concrete approaches—optimizing the existing database, migrating to a MySQL‑compatible service, and adopting a big‑data engine—detailing design, indexing, partitioning, sharding, and cloud options to restore performance.
This article explains how to extract maximum performance and scalability from PostgreSQL for multi‑terabyte workloads by leveraging specialized indexes, declarative partitioning, tablespaces, parallel query execution, read‑only replica load‑balancing, and foreign‑table sharding techniques.
This article explains how ClickHouse's MergeTree engine creates partition directories, the naming convention for partitions, and the automatic and manual merging processes that consolidate parts while tracking block numbers and merge levels.
This article explains common I/O and CPU bottlenecks in databases, introduces horizontal and vertical sharding concepts, compares sharding tools, outlines practical sharding steps, discusses typical sharding issues such as non‑partition queries and expansion, and provides a concise summary and example implementation.
This document outlines comprehensive MySQL database design standards and SQL writing guidelines, covering naming conventions, character sets, table and column specifications, index design, partitioning, data types, transaction handling, and prohibited statements to ensure stable, performant, and maintainable production databases.
The article explains how Kafka achieves high throughput by using partition‑level parallelism, sequential disk writes with segment files, extensive use of the OS page cache, zero‑copy data paths, request batching and optional compression, while also discussing the underlying disk I/O principles.
This article provides an extensive overview of Apache Kafka, covering its use cases, key concepts such as ISR, AR, HW, LEO, and LW, message ordering, the roles of partitioners, serializers and interceptors, producer and consumer client architecture, offset handling, multithreaded consumption, and topic partition management.
This article provides a thorough overview of MySQL performance tuning, covering proper table design according to the three normal forms, various index types and their creation, partitioning strategies, read/write separation, storage engine selection, and practical SQL tips for improving query efficiency.
This guide explains how to add virtual disks to a Linux VM, identify them, create partitions with fdisk, format them with XFS or ext4, mount and unmount the filesystems, and configure automatic mounting at boot using /etc/fstab.
This article presents comprehensive techniques for improving MySQL performance on large single tables, covering column choices, index design, query writing, engine selection, system parameters, hardware upgrades, read‑write splitting, caching, table partitioning, vertical and horizontal sharding, and guidance on selecting mature sharding solutions.
This article presents a thorough guide to improving MySQL performance for large single tables, covering field choices, indexing best practices, query tuning, engine selection, system parameters, hardware upgrades, read/write separation, caching layers, partitioning, vertical and horizontal sharding, and compatible scalable database alternatives.
ByteKV is a C++-based, strongly consistent, range-partitioned key‑value storage system built by ByteDance, featuring a multi‑Raft consensus layer, custom storage engines (RocksDB and BlockDB), automatic partition splitting/merging, load balancing, distributed transactions, and a SQL table layer for rich data models.
This article explains how to extend Redis beyond single‑node limits by using partitioning, master‑slave replication, Sentinel for automatic failover, and Redis Cluster with hash slots, detailing their usage, advantages, drawbacks, and configuration examples for building high‑availability and scalable in‑memory data stores.
This article explains database bottlenecks caused by IO and CPU limits, introduces horizontal and vertical sharding for databases and tables, compares popular sharding tools, discusses challenges such as distributed transactions, cross‑node joins, pagination and global ID generation, and offers guidance on when and how to apply sharding in real‑world systems.
This guide walks through essential Oracle Database administration tasks, covering how to start and shut down the database and listener, create and manage tablespaces, users, and permissions, control transactions, build and maintain indexes, views, materialized views, sequences, synonyms, and partitioned tables, with command examples and best‑practice tips.
This article explains InnoDB's logical storage hierarchy of tablespaces, segments, extents, and pages, then details MySQL's partitioning concepts, types, creation syntax, and performance considerations, providing practical examples and guidance for effective database design.
This article provides a comprehensive overview of Kafka's Producer side, covering its architecture, the ProducerRecord data structure, serialization mechanisms, partitioning logic, and the accumulator buffer, while comparing old and new Producer clients and illustrating key configurations with code examples.
When a MySQL table grows to millions of rows, query, insert, update and delete performance degrades sharply, but by applying proper field design, index strategies, SQL best practices, engine selection, system parameters, caching, partitioning, vertical and horizontal sharding, and choosing suitable client‑side or proxy‑side architectures, the database can efficiently handle tens of millions of records.
This article explains InnoDB's logical storage hierarchy—tablespace, segment, extent, page—then details MySQL partitioning concepts, types, example DDL, performance impacts, and best‑practice tips for OLTP and OLAP workloads.
This guide walks you through creating a bootable Ubuntu USB, allocating disk space, installing Ubuntu alongside Windows, adjusting the boot order, and safely removing the Linux partitions and boot entries to restore a single‑OS Windows system.
This extensive tutorial walks through Kafka producer and consumer fundamentals, demonstrates how to configure key‑based and custom partitioning, explains offset management, consumer group coordination, and essential configuration parameters, and includes complete Java code examples for real‑world e‑commerce scenarios.
Tencent Cloud CKafka powers Tencent Classroom’s pandemic‑era online teaching by replacing a custom queue with a high‑performance, highly available, partition‑based message bus that scales to millions of real‑time interactions, offers configurable replication and tuning for reliability, and integrates with big‑data and streaming tools for analytics.
This article explains the InnoDB logical storage architecture—including segments, extents, and pages—then introduces MySQL table partitioning concepts, details the main partition types (RANGE, LIST, HASH, KEY) with SQL examples, and discusses their performance implications for OLTP and OLAP workloads.
Flink’s StreamPartitioner abstracts data routing in DataStream, offering eight built‑in partitioners—including Global, Shuffle, Rebalance, KeyGroup, Broadcast, Rescale, Forward, and Custom—each with distinct channel selection logic, illustrated with source code snippets and explanations of their runtime behavior.
Data skew in Spark and Hadoop occurs when a few keys dominate shuffle traffic, causing slow tasks, OOM errors, and job failures; the article describes how to detect skew via UI metrics or sampling and offers mitigation tactics such as filtering keys, increasing shuffle partitions, custom partitioners, broadcast joins, salted keys, and Hadoop‑specific settings.
This article explains how IO and CPU bottlenecks increase active database connections, describes horizontal and vertical sharding techniques, outlines practical steps and tools for implementing sharding, and discusses common challenges such as non‑partition key queries and expansion strategies.
This article explains how to build, deploy, and continuously update event streaming pipelines using Spring Cloud Data Flow and Apache Kafka, covering common topologies, named destinations, parallel and partitioned streams, function composition, multiple input/output bindings, and practical shell commands for registration and management.
After initially implementing table sharding, we later added database sharding, sharing a detailed retrospective that covers when to partition tables, various sharding strategies (range, hash, range+hash), migration challenges, business compatibility, and final archiving practices, illustrated with diagrams and practical tips.
This article provides a comprehensive guide to MySQL performance optimization, covering database design principles, field type selection, indexing strategies, storage engine choices, query caching, partitioning methods, master‑slave replication setup, read/write splitting, load balancing, and profiling tools for diagnosing slow queries.
This guide walks through shrinking the /home partition, extending the root filesystem on a CentOS 7 server, handling busy mount issues, installing missing tools like fuser and xfsdump, resizing XFS, and safely remounting the partitions.
This article explains DBLE's hash‑based sharding algorithm, detailing how logical shards are derived via modulo, mapped to physical shards through a rule.xml configuration and mapFile, compares it with MyCat, and provides development, operational, and configuration best practices along with code examples.
This article explores practical techniques for handling massive, high‑concurrency data workloads, covering relational database limits, read/write separation, vertical and horizontal sharding, key selection, archival to NoSQL stores, and the use of heterogeneous index tables to maintain performance.
This article introduces Apache Kafka as a distributed streaming platform, explaining its three core capabilities, key concepts such as producers, topics, brokers, partitions and consumers, and detailing how messages are delivered, stored in partitions, and consumed by consumer groups.
This guide explains how to construct a client library for Youzan's customized NSQ, covering workflow configuration, nsqd discovery via nsqlookupd, connection establishment, message publishing and consumption, ordered consumption, and the implementation of message tracing with concrete command syntax and response handling.
This article demonstrates how MySQL partition tables dramatically improve query speed for large datasets, compares partition-key, secondary-index, and full‑scan queries, outlines the practical performance gains, and details the numerous limitations and trade‑offs that influence when to adopt partitioning versus sharding.
To address the long processing time caused by uneven Spark partitions when reading Kafka via the Direct approach, this article explains the SPARK‑22056 solution that modifies KafkaRDD.getPartitions to support a configurable 'topic.partition.subconcurrency' parameter, discusses its trade‑offs, and presents alternative repartition and multithreading techniques.
This article explains how OceanBase 2.0’s partitioning architecture, partition‑group feature, and a series of performance optimizations enabled Ant Group’s payment core to handle millions of transactions per second during the Double‑11 shopping festival, while also describing the system’s advantages, results, and future capabilities.
This article examines various approaches to storing massive data, including table partitioning, NoSQL/NewSQL, and database sharding, analyzes their advantages and drawbacks, and presents practical sharding designs for user, forum, and order databases, with considerations for caching, redundancy, and query routing.
A large‑scale delete operation in a MySQL PXC cluster caused transaction rollback, table locks, and flow control, leading to service slowdown, and was resolved by redesigning the cleanup mechanism with partitioned tables, stored procedures, and scheduled events.
This article explains why auto‑increment primary keys are preferred, compares B+‑tree and hash indexes, discusses MySQL partitioning, isolation levels, MVCC, and provides practical tips for designing efficient database tables and indexes.
This article explains why auto‑increment primary keys are preferred in MySQL, how B+Tree and hash indexes differ, the benefits of composite and partitioned indexes, isolation levels, MVCC, row‑level locking, triggers, stored procedures, optimization tips, MyISAM vs InnoDB, table design guidelines, and also covers Redis architecture, persistence, replication, and eviction policies.
This article presents a comprehensive overview of Oracle partitioning, covering basic concepts, the evolution of partition features across versions, practical best‑practice scenarios, and the latest enhancements in Oracle 12c, 12.2, and 18c that improve performance, manageability, and availability.
This article reviews common MySQL pitfalls—including engine choice, table partitioning, index design, slow‑query causes and optimizations, diagnostic tools, and how to recover from accidental data loss—offering practical guidance for developers to improve performance and reliability.
This article explains how to choose MySQL storage engines, apply effective table partitioning and indexing, identify common causes of slow queries, adopt practical SQL optimization tips, use diagnostic tools, and handle data loss or bugs with proper recovery procedures.
This article walks through a real‑world monitoring project that must store over 400 million rows per day in SQL Server 2012, detailing the performance bottlenecks, bulk‑copy tuning, schema redesign, partitioning strategies, index removal and creation, and query‑optimisation techniques that finally achieve sub‑second query times and meet strict latency requirements.
Database sharding, or splitting databases into multiple instances and tables, is essential for scaling backend systems; this article explains its importance, historical context, core concepts, storage stretching techniques, index and consistency handling, and provides a practical design example for transaction systems.