A developer used INSERT‑INTO‑SELECT to migrate millions of rows, unintentionally locking the source table and causing massive payment failures, but adding an index and forcing its use resolved the issue and restored safe migration.
This article explains the motivations for moving a high‑volume operation‑log system from MongoDB to Elasticsearch, outlines the existing architecture, details capacity planning, index design, and a step‑by‑step migration process using Kafka, DataX, and Spring Boot, and shares the performance gains and lessons learned.
This article details how 58.com built a massive Hadoop‑based offline computing platform with over 4,000 servers and hundreds of petabytes of storage, addressing scaling, stability, GC, YARN scheduling, SparkSQL migration, storage operations, and a large‑scale cross‑datacenter migration.
This guide introduces dtle, an open‑source MySQL data‑transfer middleware, covering download, installation on three nodes, detailed dtle.conf configuration, startup commands, creating source and target users, preparing job.json, using HTTP API to create and monitor migration jobs, and verifying data migration results.
Facing massive MySQL tables that take days to alter, 360's cloud team migrated to TiDB and discovered instant online DDL, fast column additions, efficient COUNT(*) queries, seamless MySQL‑to‑TiDB migration, and a suite of tuning knobs that keep disk usage low and performance high across billions of rows.
This article outlines a comprehensive methodology for migrating small‑mainframe platforms—including hardware assessment, solution design, implementation steps, risk evaluation, and three common data‑migration techniques—so that businesses can safely transition workloads to modern x86 servers while preserving data integrity and service continuity.
This article shares the experience of deploying TiDB on the 360 Cloud Platform, covering background challenges with massive tables, TiDB's online DDL and high‑performance features, cluster architecture, DM migration workflow, common operational issues, and detailed tuning parameters to improve latency, QPS, and resource utilization.
This article explains the root causes of MySQL character‑set garbling, illustrates the encoding/decoding steps during data insertion and retrieval, and provides practical methods—including dump‑reload and binary conversion—to prevent and repair corrupted data.
This article details Ctrip's cross‑datacenter Hadoop architecture, covering the evolution of its Hadoop platform, the challenges of multi‑site bandwidth and latency, design choices between multi‑cluster and single‑cluster solutions, and the concrete HDFS, YARN, balancer, migration, monitoring, and throttling implementations that enable transparent, consistent, and efficient multi‑datacenter operations.
The 2018 TSB bank migration disaster, which corrupted data for 13 million customers and cost billions, illustrates how inadequate testing, overly complex system integration, and poor change‑management practices can cripple banking IT operations and highlight the need for rigorous operational risk controls.
The article examines the disastrous 2018 TSB bank data migration, detailing how rushed system replication, inadequate testing, and complex micro‑service architecture caused massive data loss, regulatory fallout, and costly remediation, while highlighting broader lessons for IT operations and resilience in financial services.
The article describes the evolution from the legacy XDATA tool to the new XFlink system, detailing its architecture, core plugins, parser and deployment modules, resource management with Yarn, monitoring via Prometheus and Grafana, and planned enhancements such as Flink SQL configuration and modular plugins.
This article explains how to reverse the typical data flow by extracting massive Rowkeys from HBase with MapReduce, storing them on HDFS, and then using batch Get operations to retrieve the full records and write them into Kafka, while handling retries and monitoring progress.
This article details how Zhihu’s Moneta service tackled massive data growth—over a trillion rows—by migrating from MySQL sharding to the HTAP‑capable TiDB platform, describing architectural changes, performance gains, migration lessons, and future expectations with TiDB 3.0.
The article describes how to migrate a multi‑gigabyte InnoDB table from a pre‑production MySQL 5.7 instance to production using the transportable tablespace feature, detailing the step‑by‑step procedure, performance benefits, and the specific limitations of this method.
This article explains how Alibaba Cloud's BDS migration service enables continuous, high‑performance migration of HBase clusters—including schema, full data, and incremental sync—across version upgrades, hardware changes, network migrations, and cross‑region scenarios, while ensuring stability and minimal impact on live workloads.
FastLoad, Didi’s one‑click DTS platform, accelerates migration of terabyte‑scale offline data into its Fusion storage by using RocksDB’s IngestFile to import SST files directly, cutting a 1 TB load from twelve hours to one, while supporting thousands of daily tasks with 99.99% stability.
This comprehensive guide introduces Apache Phoenix as an HBase SQL layer, covering quick‑start steps, supported data types, DML syntax, salted tables to prevent hotspots, various secondary index types, bulk‑load methods, auto‑increment IDs, dynamic columns, pagination, query plan analysis, and data migration techniques.
This article chronicles a Java backend data‑migration project where the author transformed a two‑month, 2000‑million‑record process into a four‑hour job by iteratively redesigning the architecture from a monolithic procedural approach to a fully decoupled, multithreaded, queue‑driven system, highlighting key performance bottlenecks and solutions.
Youku’s 2017 migration from an on‑premises Hadoop cluster to Alibaba Cloud MaxCompute delivered a unified, elastic data pipeline that cut compute and storage costs by roughly half, handled billions of daily log records, boosted performance and scalability, and empowered analysts with self‑service tools and a rich ecosystem.
This article reviews the complete process of database and table sharding, covering when to shard, range and hash strategies, combining both, data migration challenges, business compatibility, and long‑term archiving to keep MySQL workloads manageable.
This article explains how the WTable distributed key‑value store leverages RocksDB’s LSM‑tree architecture and slot‑based data distribution to redesign its scaling workflow, separating full and incremental data migration to reduce compaction overhead and achieve high‑speed, low‑impact cluster expansion.
This article explains how Youku processed billions of daily logs, migrated from Hadoop to Alibaba Cloud MaxCompute in 2017, and achieved lower compute and storage costs, faster data delivery, and greater operational flexibility through a robust big‑data platform tailored to its complex business needs.
This article chronicles a step‑by‑step performance overhaul of a 20‑million‑record migration project, detailing four architectural revisions—from a single‑threaded procedural script to a fully decoupled, multithreaded, interface‑driven solution—that reduced processing time from two months to just four hours while ensuring data consistency, recoverability, and scalability.
This article provides a comprehensive step‑by‑step guide for migrating data from Oracle to MySQL using Oracle GoldenGate, covering architecture overview, environment setup, table and data migration, configuration of source and target processes, handling of common issues, data validation, and reference resources.
This tutorial explains how to use Navicat to back up a test database, delete and restore product data, and synchronize schema changes between test and production databases, providing detailed step‑by‑step instructions with screenshots.
This article describes the challenges of scaling Qunar's Hadoop NameNode, introduces HDFS Federation and the FastCopy tool, presents performance tests comparing FastCopy with DistCp, and details the development and evaluation of an optimized qFastCopy solution that reduces multi‑petabyte migration time from hours to a few.
This article details Didi's evolution from a self‑built NoSQL system called Fusion to a NewSQL solution and outlines the challenges, architectural highlights, data migration mechanisms, and future distributed database design, providing concrete metrics, diagrams, and practical lessons learned.
OpenSDS Gelato is an open‑source multi‑cloud data management platform that unifies object storage across public clouds and on‑premises, offers S3‑compatible APIs, supports backend pooling and policy‑driven cross‑cloud data flow, and outlines a roadmap for lifecycle management and advanced migration tools.
This article provides a comprehensive technical guide on HBase, covering its core concepts, advantages and drawbacks, architecture layers, practical use cases, and a detailed step‑by‑step process for large‑scale cross‑datacenter migration using snapshot‑based strategies, with commands, diagrams, and lessons learned.
In this talk, Tencent Cloud senior engineer Zou Peng outlines cloud‑native best practices for Redis 4.0 clusters, detailing a smart‑proxy architecture with gossip‑based management, lazy‑scan big‑key handling, automatic fail‑over, the high‑performance CKV engine, and practical guidance on scaling, sharding, monitoring, and cost‑effective high‑availability deployment.
This article summarizes the current usage of HBase at Ping An Technology, the challenges it addresses, detailed client‑ and server‑side performance and stability optimizations, high‑availability mechanisms, data migration strategies, monitoring and repair practices, and future development plans.
iQIYI migrated its critical Edge Control, Video Transcoding, and User Login services from MySQL to TiDB, gaining automatic sharding, high‑availability multi‑datacenter replication, and stable query performance that eliminated storage bottlenecks, complex sharding logic and frequent downtime, while enabling future OLTP/OLAP integration.
After struggling with a two‑month data migration that processed 20 million users, the author iteratively redesigned the system—from a single‑threaded, procedural approach to a fully decoupled, multithreaded architecture using queues and batch operations—ultimately reducing runtime to just four hours while ensuring data consistency and recoverability.
This article describes the background, analysis, solution selection, architecture redesign, execution steps, and results of a large‑scale order table splitting project that separates hot and cold data, applies horizontal sharding, and uses MySQL partitioning to control table size and boost query performance.
This article walks through diagnosing a MySQL crash, using innodb_force_recovery to start the server, exporting the data with mysqldump, handling charset mismatches, importing into a new instance, and restoring normal operation while offering best‑practice backup recommendations.
This article examines the architectural redesign of Alibaba's High‑Performance Time Series Database (HiTSDB), covering storage model changes, inverted‑index enhancements, a pipelined streaming aggregation engine, data‑migration strategies, and performance benchmarks that together deliver over tenfold query speed improvements.
This article shares practical experience deploying MariaDB ColumnStore for offline OLAP, covering its MPP‑based architecture, user‑experience benefits, high‑availability options, numerous syntax and data‑type limitations, and detailed scripts for schema conversion and incremental data extraction using pt‑archiver.
This guide answers the most common questions about Inceptor, covering its purpose, installation, command‑line interaction, table creation, partitioning, execution modes, error handling, column alteration, query planning, data migration, and CSV import settings.
A DBA on call discovers a sudden traffic spike caused by a few massive hot keys on a storage server, quickly isolates the issue, migrates data, applies throttling and caching, and outlines automation ideas to prevent future overloads, illustrating practical database operations and incident response.
This article examines Suning's evolution of its membership platform—from an early offline POS system to a vertically split, cloud‑native architecture—detailing capacity planning, performance testing, data migration with Spark, multi‑active deployment, and future plans for cross‑region high availability.
The article details Suning’s decade‑long evolution of its membership platform for the Double‑11 shopping festival, covering early offline CS architecture, the transition to a WebSphere‑based e‑commerce system, vertical module splitting, data migration with Spark and Hive, and the implementation of same‑city multi‑active deployment.
This article details how Vimur, a Kafka‑based real‑time data pipeline, addresses the challenges of service splitting, database sharding, data migration, and synchronization by using CDC, a unified Avro format, and a change distribution platform to support search indexing, cache refresh, and reactive architectures.
This article details Grab's migration from a single‑node Redis to AWS Elasticache, outlining the evaluated solutions, the chosen architecture for horizontal read scalability, and a six‑step migration process that ensured safe, zero‑downtime transition.
This article provides a detailed introduction to data migration, explaining why and when it is needed, outlining its major challenges such as security, compatibility and downtime, and describing the four primary migration approaches—application, volume‑manager, network, and storage—along with common tools and technologies.
When moving data from Oracle to MySQL, Chinese characters may appear as garbled text because the server uses a latin1 charset; this guide explains three levels of fixes—table, database, and server—by configuring UTF‑8 settings and verifying the changes.
The migration of 500 GB HBase and 5 TB Solr data from version 0.94.15 to 1.2.1 required fixing hardware clock drift, DNS hostname issues, and missing Snappy support, and demonstrated that a brute‑force HDFS transfer is more reliable than import/export when handling deprecated parameters.
Tencent’s senior engineer explains how the TDW (Tencent Distributed Data Warehouse) grew from a few hundred to thousands of nodes, the challenges of cross‑city migration, and the modeling, relationship‑chain, dual‑write tables, and platform strategies they built to ensure seamless, low‑impact data and task migration.
This article compiles frequently used MySQL SQL commands—including data migration, multi‑table updates, conditional updates, aggregation inserts, and various ALTER TABLE operations such as adding indexes, dropping and adding columns—to serve as a practical reference for database developers.
This article explains how a massive order database exceeding 200 GB was vertically and horizontally sharded, introduces query, range, and hash sharding strategies, presents a custom timestamp‑based unique ID scheme, and outlines a multi‑phase data migration plan to achieve long‑term scalability.
This article presents a comprehensive, practice‑driven analysis of horizontal database sharding for a high‑traffic e‑commerce order system, covering sharding dimensions, strategies, quantity planning, transparent routing, pagination challenges, lookup mapping, overall architecture, deployment steps, and the measurable performance and cost benefits achieved.
This guide explains how to clean redundant fields, restructure a massive MySQL table, batch‑fetch data with LIMIT, rewrite slow queries, and compare three insertion methods—including prepared statements and bulk inserts—to cut migration time from hours to under 20 minutes.
An automated database operations platform shifts routine DBA tasks to developer self‑service and fully automates processes such as cluster provisioning, scaling, backup, migration, and sharding, using a stateless workflow center, job queue, and unified SqlEditor to improve efficiency, safety, and auditability.
This article details JD's One‑Yuan Grab Treasure platform redesign, covering business growth drivers, database sharding estimation, hash‑plus‑range routing implementation, Elasticsearch aggregation, Canal‑based sync, historical data migration, and downgrade mechanisms to ensure high‑throughput, reliable order processing during massive sales events.
This article explains why data migration is essential for maintaining secure and reliable data services, outlines the typical triggers such as equipment end‑of‑life, and details the major challenges—including security, compatibility, downtime, and required expertise—while reviewing the four primary migration approaches and their practical implementations.
This tutorial details the complete process of moving a single‑node DB2 v9.7 PHSDB database to K‑DB11G, covering schema‑to‑user mapping, object conversion, constraint handling, identity column emulation, manual adjustments for views/functions, and the export‑import scripts required for data migration.
This article explains how a rapidly growing order database exceeding 200 GB can be scaled using vertical and horizontal partitioning, various sharding strategies, custom ID generation, migration phases, and practical considerations for transactions and complex queries.
The article outlines a comprehensive database sharding solution for a massive order system, describing vertical and horizontal partitioning, various sharding strategies, a custom ID scheme, transaction handling, complex query constraints, and a three‑stage data migration plan to achieve long‑term scalability.
This guide explains why the official Redis Cluster tools lack password support, outlines the security risks of an unauthenticated cluster, and introduces a custom management utility that adds password authentication, automates slot migration, and simplifies adding or removing nodes, complete with step‑by‑step testing procedures.
This article analyzes YouPai.com's evolution from a simple master‑slave setup to horizontal sharding using a mapping table, detailing migration steps, access flow, and challenges such as cross‑database joins, consistency, and ID generation.
This article explains MySQL architecture design for 24/7 online services, covering business analysis of read‑write patterns, common deployment models, consistency levels, data migration, sharding strategies (vertical, horizontal, and business‑level), and cluster scaling for high availability.
This article explains the four‑step online data migration process—including dual‑write, historical data transfer, read cut‑over, and cleanup—using a real‑world MySQL‑to‑HBase fan‑list case study, and provides practical tips for ensuring zero‑downtime and high data consistency.
This article presents a comprehensive side‑by‑side mapping of 45 common SQL Server functions to their Oracle equivalents, covering numeric, string, date, and miscellaneous operations with example queries and notes on usage differences.
