This guide reviews the most popular ETL and data integration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their core features, architectures, and typical use cases to help you choose the right solution for data migration and synchronization.
This article outlines the essential steps for validating data after migration, discusses common issues and their solutions, and recommends tools and a Python script for comparing source and target data to ensure integrity, accuracy, and consistency.
This article describes the challenges of scaling a multi‑system operation log platform, proposes a new unified log schema, compares non‑intrusive and intrusive collection approaches using Canal and AOP/annotations, and outlines a rule‑engine‑driven migration strategy for legacy log data.
This article outlines a two‑stage migration strategy—application splitting and data splitting—detailing how to isolate APIs, message queues, and scheduled tasks, manage dual‑write database synchronization, ensure gray‑scale traffic routing, and perform validation to achieve a reliable, low‑risk transition for large‑scale backend systems.
This guide walks through essential Redis operations, covering real‑time monitoring with the INFO command and Prometheus‑compatible exporters, data migration using Redis‑shake, consistency verification via Redis‑full‑check, and comprehensive cluster management with CacheCloud, providing practical tools for reliable Redis administration.
This guide explains how to use the open‑source binlog_parse_sql tool to parse MySQL 8.0 binary logs, generate SQL statements, and seamlessly migrate data to MariaDB or ClickHouse, covering installation, execution modes, troubleshooting, and practical ETL scenarios.
The article describes how a finance team tackled a 50‑million‑row MySQL transaction table by analyzing the pain points, defining split goals, selecting sharding‑jdbc, designing multi‑source transaction handling, pagination across shards, data migration strategies, and a staged rollout to ensure stability and performance.
Facing a MySQL transaction table exceeding 50 million rows and growing rapidly, a finance team redesigned the schema using sharding‑jdbc, chose transaction time as the shard key, tackled multi‑data‑source transactions and cross‑shard pagination, and executed a phased migration to ensure stability and performance.
iQIYI accelerated its big‑data platform by migrating the OLAP layer from Hive to Spark SQL, achieving a 67 % speedup, 50 % CPU reduction and 44 % memory savings, while automating the conversion of tens of thousands of tasks and delivering faster analytics for advertising, BI, membership and user‑growth services.
Facing a 50‑million‑row financial transaction table, the team implemented horizontal sharding with sharding‑jdbc, designed migration and pagination strategies, tackled multi‑datasource transaction issues, and outlined a phased rollout, providing practical insights for large‑scale MySQL table splitting and backend system stability.
This article presents a comprehensive engineering case study of a high‑traffic membership system, detailing the dual‑center Elasticsearch high‑availability design, traffic‑isolated three‑cluster ES architecture, Redis caching strategy, dual‑center MySQL partitioning and migration plan, abnormal member relationship governance, and future fine‑grained flow‑control and downgrade policies.
This article details how a finance team tackled a 50‑million‑row MySQL transaction table by designing a horizontal sharding strategy with Sharding‑JDBC, addressing performance bottlenecks, multi‑source transaction handling, pagination across shards, and a phased data migration and rollout plan.
The article details how a finance team tackled a 50‑million‑row MySQL transaction table by analyzing the pre‑split problems, defining split goals, choosing sharding‑jdbc, addressing multi‑datasource transaction and pagination issues, designing a phased migration and rollout plan, and summarizing lessons learned.
This article examines the challenges of data consistency verification during database migrations, reviews existing tools like pt-table-checksum, sync‑diff‑inspector and gt‑checksum, outlines essential capabilities, and proposes a detailed workflow and future enhancements for a versatile, low‑impact consistency checking solution.
This article reviews eight widely used ETL and data‑migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their core features, architectures, supported data sources, and typical usage scenarios to help practitioners choose the right solution.
This article explains what ClickHouse is, why a large MySQL table caused multi‑minute queries, and how migrating the data to ClickHouse using Docker, CREATE TABLE AS SELECT, and a lightweight sync tool reduced query time to under one second, while highlighting pitfalls and performance reasons.
This article introduces ClickHouse as an OLAP columnar database, explains its advantages over row‑based systems, details a real‑world migration from MySQL, shows performance benchmarks, shares practical installation and data‑sync methods, and highlights common pitfalls and solutions.
This article explains common business scenarios such as large‑table splitting, cross‑database migration, and data synchronization, then compares loss‑bearing and loss‑less migration strategies, and provides detailed step‑by‑step implementation guidance for smooth dual‑write migration and real‑world tooling.
This article provides a detailed analysis of the OMS community edition 3.3.1 architecture, explains each internal process, describes the migration workflow and monitoring metrics, and offers practical optimization tips to improve data migration performance.
This guide reviews eight popular ETL and data migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their core features, architectures, and use cases to help engineers choose the right solution for reliable data integration.
Explore a comprehensive overview of eight popular ETL and data migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their features, architectures, and use cases to help you choose the right solution for efficient data integration.
This article presents a detailed case study of Gaotu's migration of its on‑premises big‑data platform to Tencent Cloud, covering background challenges, a phased migration plan, implementation lessons, cost‑control measures, and future optimization strategies, with practical insights for similar enterprises.
This guide walks through a collection of practical MySQL functions and commands—including GROUP_CONCAT, CHAR_LENGTH, LOCATE, REPLACE, NOW, INSERT…SELECT, INSERT…IGNORE, SELECT FOR UPDATE, ON DUPLICATE KEY UPDATE, SHOW CREATE TABLE, CREATE TABLE … SELECT, EXPLAIN, SHOW PROCESSLIST and mysqldump—providing clear syntax, example queries, result screenshots and key usage notes to help developers write more efficient and reliable SQL.
Facing massive user and order data growth, this article examines storage options, compares relational, NoSQL, and NewSQL databases, and provides a detailed guide to implementing MySQL‑based sharding with proxy and client modes, key selection, partition strategies, migration steps, and future scaling considerations.
This article presents a step‑by‑step practice of migrating massive billing data to 32 sharded databases with Sharding‑Proxy, covering background, objectives, four migration solutions, detailed proxy installation and configuration, debugging, migration workflow, data validation, common issues and their resolutions.
This guide explains how to scale a sharded database horizontally by adding new nodes, handling data migration with stop‑service, stop‑write, log‑based, dual‑write, and smooth 2N strategies, and implementing high‑availability using MariaDB double‑master replication and Keepalived, complete with configuration examples and code snippets.
This guide explains why a single massive DELETE on a 500‑million‑row table creates long‑running transactions and locks, and shows step‑by‑step techniques—date‑range batching, primary‑key range slicing, and insert‑instead‑of‑delete—to break the operation into manageable chunks and improve performance.
This article examines how vertical and horizontal data partitioning leads to migration and consistency challenges, outlines full, full‑plus‑incremental, and binlog‑based migration strategies, and compares strong‑consistency XA transactions with flexible BASE approaches such as TCC and AT, including their components, workflows, advantages, and drawbacks.
This article outlines common data migration scenarios such as system consolidation and database sharding, details analysis of user data before migration, discusses conflict resolution rules, presents migration planning and acceptance testing steps, and highlights post‑release monitoring and user feedback handling.
This article introduces essential Redis operational tools, covering real‑time monitoring with the INFO command and Prometheus‑exporter, data migration using Redis‑shake, consistency checking via Redis‑full‑check, and cluster management through CacheCloud, providing practical guidance for administrators.
This article explains PostgreSQL logical replication concepts, key components like replication slots and output plugins, common challenges such as failover slots, DDL sync, and bidirectional loops, and demonstrates practical full‑and‑incremental data migration to the cloud.
This guide explains how to move a self‑hosted ZooKeeper cluster to Alibaba Cloud MSE ZooKeeper by exporting the snapshot file, preparing the MSE instance, uploading the snapshot, and verifying the migration with example client code.
The article presents a transparent data‑encryption framework for database security that uses an AOP‑based driver interceptor to rewrite SQL, encrypt/decrypt sensitive fields, manage performance with locks and caching, support configurable algorithms via SPI/YAML, and guide a three‑phase migration with minimal code intrusion.
This article examines the massive data, cost, stability, and diversity challenges of the AI‑driven, big‑data, cloud‑first "ABC" era and explains how Baidu's Canghai storage portfolio—including BOS, CDS, CFS, PFS, RapidFS, CloudFlow, and storage gateways—addresses each issue through scalable architecture, tiered lifecycle policies, multi‑AZ disaster recovery, and integrated hybrid‑cloud solutions.
This article outlines the four major storage challenges of the ABC era—massive scale, cost efficiency, stability, and diversity—and explains how Baidu’s Canghai storage suite, including BOS, CDS, CFS, PFS, RapidFS, CloudFlow, and storage gateways, addresses each through multi‑cloud migration, tiered lifecycle management, and robust disaster‑recovery solutions.
This article provides a comprehensive, code‑rich guide on using TiDB DM v2.0.7 to perform full and incremental migration of terabyte‑scale MySQL data to DBLE, covering environment setup, TiDB DM installation, DBLE configuration, data generation, task execution, verification, and important operational considerations.
The article outlines a proxy‑based migration framework that enables stable system architecture upgrades by using total and gray‑scale switches to achieve gradual rollout, rollback, non‑intrusive business logic, and graceful shutdown, while handling single‑query routing, multi‑query deduplication, double‑write updates, and direct inserts, illustrated with AspectJ‑driven code and retail store‑ID examples.
The article details Bilibili's evolution of its offline computing platform from Hadoop‑based Hive to Spark, describing migration tools, SQL conversion, result and resource comparison, shuffle stability, small‑file handling, runtime filters, data skipping, ZSTD support, Hive Metastore federation, traffic control, and future optimization directions.
This article describes how a finance team tackled a 500 million‑row MySQL transaction table by analyzing pre‑split issues, defining sharding goals, selecting sharding‑jdbc, addressing multi‑datasource transaction and pagination challenges, designing a hybrid data‑migration plan, and executing a three‑stage rollout to ensure system stability and performance.
The paper proposes a Bloom‑filter‑based deduplication service for short‑video recommendation that moves three‑month playback histories to disk‑backed Bloom filters while keeping the latest 100 served IDs in Redis, employing write batching, sharding, expiration policies, and an incremental migration strategy to replace memory‑intensive Redis ZSets and dramatically reduce storage costs.
This guide explains three Elasticsearch migration approaches—COS snapshots for large datasets, Logstash for flexible full or incremental transfers, and elasticdump for small‑scale migrations—detailing setup steps, version compatibility, command examples, and when to choose each method.
This article details how ZhaiZhai tackled severe order database performance and capacity issues by adopting ShardingSphere for sharding, describing the challenges, mitigation steps, shard key design, migration process, and the significant performance improvements achieved.
This article details a real‑world cloud migration journey using Tencent Cloud services, explains one‑click provisioning of components like Zookeeper and Redis, outlines data migration strategies, and demonstrates how to automate build and deployment pipelines with Ansible playbooks and scripts.
This article details the common pitfalls encountered when synchronizing Oracle databases to AnalyticDB PostgreSQL, covering null versus empty string handling, data type conversion challenges, character set issues, special character processing, and comprehensive testing strategies to ensure data consistency and performance.
This article describes how Ant Group tackled the performance ceiling of its massive Sigma Kubernetes clusters by horizontally splitting etcd storage for Pods, Leases and Events, redesigning watch handling to avoid component restarts, and using snapshot‑based migration to preserve data integrity while reducing latency.
This article describes how a Python‑based diff utility was created to compare old and new API responses during a promotion change‑record migration, detailing its implementation, handling of JSON structures, improvements over the first version, and plans for integration with an API testing platform.
This article introduces essential Redis operational tools, covering the INFO command for monitoring, Prometheus‑based redis‑exporter visualization, the Redis‑shake data migration utility, Redis‑full‑check consistency verification, and the CacheCloud platform for comprehensive cluster management.
This article introduces essential Redis operational tools, covering the INFO command for real‑time monitoring, Prometheus‑based exporters, data migration utilities like Redis‑shake, consistency checking with Redis‑full‑check, and cluster management via CacheCloud, while offering practical usage tips.
This article details the background, challenges, and step‑by‑step solutions for migrating over a hundred petabytes of Hadoop HDFS data across data centers within a month, covering strategy selection, code modifications, balance optimization, and tool enhancements.
Zhihu’s Moneta service, handling over a trillion rows and billions of daily writes, migrated from MySQL to TiDB, achieving millisecond query latency, high availability, and horizontal scalability, and the article details the architecture, performance metrics, migration challenges, and lessons learned from this large‑scale deployment.
This article walks through a practical approach to safely split a 50 million‑row MySQL history table, add missing indexes, and migrate data in batches using backup, table‑copy, and incremental INSERT operations while avoiding long locks and performance degradation.
This article summarizes the author's extensive experience with microservice adoption, covering its benefits, the challenges of stability, service degradation strategies, distributed transaction patterns, data migration techniques, and practical monitoring using APM tools to help teams successfully transform to microservices.
This article introduces ClickHouse, a high‑performance open‑source columnar database, explains its architecture versus row‑based systems, details migration from MySQL, showcases installation, performance benchmarks, data‑sync strategies, common pitfalls, and summarizes its benefits for large‑scale analytical workloads.
Kettle, a pure‑Java open‑source ETL tool (Chinese name "Shuihu"), enables efficient, installation‑free data extraction and synchronization, allowing you to copy tables or views from a SQL Server database directly into a MySQL table using a straightforward workflow.
This article introduces ClickHouse as a column‑oriented OLAP database, compares it with traditional row‑based databases, and details a real‑world migration from MySQL that reduced query time from minutes to under one second, along with performance testing, sync strategies, and common pitfalls.
When a Kafka cluster reaches load limits or experiences sudden traffic spikes, urgent expansion is needed, but data migration can be time‑consuming and risky; this guide outlines several practical techniques—including adjusting retention, adding partitions, leader switching, and single‑replica operation—to quickly scale clusters while minimizing data movement and service disruption.
This article demonstrates how to export full data from TiDB to MySQL and vice‑versa using TiDB's dumper/dumpling tools, MySQL's mysqldump, and the tidb-lightning importer, covering both SQL and CSV formats, scripts for automation, and verification steps for large tables.
This article details Maoyan's multi‑account integration journey, describing the background, challenges, industry research, and a three‑phase write‑through solution that merges APP and mini‑program accounts using phone numbers as a common anchor while ensuring data consistency, scalability, and minimal user impact.
This article outlines a practical method for migrating MySQL tables with millions of rows by exporting data to an outfile, compressing, transferring, and loading it on the target server, while also addressing strict SQL mode errors that can arise during import.
This article details a practical MySQL data migration workflow for moving a million‑row table to a new server, comparing mysqldump, custom Go scripts with Redis pipelines, and direct file‑based import/export, and provides step‑by‑step commands, configuration tips, and performance considerations.
This article presents a comprehensive guide to synchronizing heterogeneous data sources with ElasticSearch, migrating clusters across environments, and implementing robust disaster‑recovery solutions for both intra‑city and inter‑city high‑availability scenarios.
This article explains how to resolve incompatibility between SyBase's default datetime export format and MySQL's datetime format by using the convert() function to reformat dates and str_replace() to adjust delimiters, then exporting the data via bcp for seamless import.
This article describes how NetEase developed a Hive metadata migration and merging tool that consolidates metadata from multiple independent Hive clusters into a single Hive metastore without moving HDFS data, detailing the challenges, ID handling, database operations, and step‑by‑step migration process.
This article explains the signs that indicate a legacy system needs refactoring, describes the challenges of a monolithic architecture with massive tables, outlines goals such as zero business impact and migration to microservices, and details a step‑by‑step migration strategy using data pipelines and careful interface compatibility.
This article shares practical experience with TiDB Data Migration (DM), covering its background, architecture, key features, online DDL support, common error handling such as duplicate‑key issues, large‑scale import tuning, configuration limits, and cleanup recommendations for reliable MySQL‑to‑TiDB synchronization.
This article introduces ClickHouse, explains its column‑oriented architecture versus row‑oriented databases, details a MySQL‑to‑ClickHouse migration, shows a performance benchmark that reduces query time from minutes to seconds, and shares practical deployment tips and common pitfalls.
This article explains how to split large-scale database tables using vertical and horizontal partitioning, compares hash-modulo and range-based sharding, and proposes a hybrid approach that leverages both methods to balance data distribution, avoid hotspot issues, and eliminate the need for costly data migrations during scaling.
This article recounts a costly MySQL data‑migration failure caused by an unchecked INSERT INTO SELECT that triggered full‑table scans, lock contention and out‑of‑memory errors, and explains how proper indexing and awareness of transaction isolation can prevent such incidents.
An engineer’s costly mistake using MySQL’s INSERT INTO SELECT for nightly data migration led to out‑of‑memory crashes, full‑table scans, and payment record loss, prompting a deep dive into locking behavior, transaction isolation, and how proper indexing can safely rescue large‑scale inserts.
This article details the end‑to‑end process of expanding a high‑traffic ride‑hailing order system on Alibaba Cloud, covering capacity planning, database sharding from 256 to 4096 tables, custom data‑sync middleware, validation, repair, and a gray‑release migration that achieved zero incidents.
This article outlines practical techniques for multi‑cloud data migration, covering challenges, resource categories, migration phases, and detailed synchronization and verification methods for MySQL databases, file systems, and object storage using tools such as UDTS, pt‑table‑checksum, sync_diff_inspector and US3SYNC.
This article explains how RedisSyncer enables fast, reliable cross‑version and cross‑cluster Redis data migration in multi‑cloud environments, addressing challenges such as version mismatches, node inconsistencies, and rapid rollback while improving access speed and reducing database load.
This case study explains how a client with a 2‑billion‑row MySQL table avoided disk exhaustion and poor query performance by streaming data through Kafka to Google BigQuery, partitioning the data, and cleaning it to reclaim storage—all without downtime.
This article explains the challenges of cross‑cloud migration, outlines three resource categories, details MySQL, file and object storage synchronization methods, presents practical tools and validation techniques, and shares real‑world case studies to ensure data integrity and efficiency.
This article explains Android's scoped storage model introduced in Android 10, details how it changes file access, provides migration strategies for new and legacy data, and includes practical code examples for handling media files, legacy storage flags, and common file‑operation errors across Android versions.
This guide walks through migrating documents from a MongoDB 3.4.1 cluster to an Apache CouchDB 2.2 cluster using a Python script that leverages pymongo and CouchDB’s REST API, covering environment setup, migration steps, code execution, and verification of transferred data.
This guide explains three Redis migration methods, focuses on using the third‑party tools redis‑dump and redis‑load, and provides detailed steps to upgrade Ruby, install RVM, set up redis‑dump, and export/import data in JSON format.
This article details a real‑world solution for migrating more than two billion MySQL records to Google BigQuery by streaming data through Kafka, employing partitioned tables, data filtering, and incremental migration to avoid downtime and reduce storage costs.
Facing a MySQL table with over 2 billion continuously growing rows, the team designed a cloud‑based solution using Kafka to stream data into Google BigQuery, applied partitioned schemas and data cleaning to improve query performance, reduce storage costs, and avoid downtime.
This article introduces ClickHouse as a column‑oriented OLAP database, explains its architectural differences from row‑based systems, details a real‑world migration from MySQL with code examples, showcases performance gains, and shares practical tips and pitfalls for production use.
This article explores common data migration scenarios such as sharding, storage engine changes, and system switches, and outlines practical approaches for bulk and incremental migration, validation techniques, cut‑over strategies, and ID management to ensure reliable, low‑risk database transitions.
TiDB is a distributed NewSQL database that combines MySQL compatibility with a stateless SQL layer, a PD metadata manager, and a transactional key‑value store, offering automatic scaling, multi‑data‑center Raft replication, global secondary indexes, Percolator‑style 2PC, HTAP via TiFlash, and a rich ecosystem of migration, backup, CDC, and monitoring tools, as demonstrated by its large‑scale deployment at DeWu.
This article provides a step‑by‑step guide to building an NPM private registry using Cnpmjs.org, covering containerized deployment with Docker, migrating packages from Verdaccio, implementing OSS disaster‑recovery backups, configuring email notifications, and tips for further customization.
This article documents a step‑by‑step MySQL data migration case involving 140 million rows, covering export via SELECT INTO OUTFILE, import with LOAD DATA INFILE, and a series of performance tweaks such as index removal, session variable tuning, engine selection, and monitoring to achieve sub‑10‑minute load times.
This article shares a comprehensive overview of TiDB Data Migration (DM), covering its architecture, configuration, online DDL support, common pitfalls such as duplicate‑key errors, large‑scale import tuning, version limits, and cleanup recommendations to help DBAs efficiently migrate MySQL/MariaDB workloads to TiDB.
This article summarizes a JD Cloud‑TiDB live webcast, explaining why JD Cloud chose TiDB, detailing its distributed architecture (TiDB Server, TiKV storage, PD), highlighting features such as horizontal scaling, strong consistency, real‑time analytics, backup, migration, and showcasing multiple enterprise migration case studies that demonstrate TiDB’s ability to overcome traditional database bottlenecks.
This guide outlines the three‑stage data‑migration testing process—preparation, execution, and verification—detailing how to assess data scope, choose migration methods, evaluate system compatibility, construct test scenarios, validate scripts, monitor performance, and confirm backend, frontend, and business‑logic integrity after migration.
This guide walks you through downloading, extracting, compiling, and installing Redis from source on Ubuntu, configuring its files and startup scripts, securing the service, enabling automatic boot start, and performing data migration with detailed commands and examples.
This article explains how Vivo uses Alibaba's open‑source Canal to capture MySQL binlog changes, achieve non‑stop sharding and cross‑region data migrations, ensure high availability with Zookeeper, and share practical lessons on serialization, consistency, and monitoring in large‑scale backend systems.
This article chronicles Tencent's journey from the early development of the TFS distributed storage platform to large‑scale data migrations, flexible bandwidth strategies, and the creation of the cloud‑native YottaStore, illustrating how a small architecture team solved massive storage challenges for billions of users.
The team migrated a large, multi‑service microservice publishing platform—including Xianyu, Taobao, Alipay, and Tmall—from a centralized environment to a department‑built infrastructure in ten working days by cloning the repo, updating configurations, separating databases, rigorously verifying functionality across dev, pre‑release, and production, and ensuring isolation and monitoring for stability.
This article introduces ClickHouse, compares column‑store and row‑store databases, shows how migrating a 50‑million‑row MySQL table to ClickHouse reduced query time from minutes to under one second, and shares practical installation, migration, performance testing, and synchronization tips.
This article introduces ClickHouse, compares column‑ and row‑oriented storage, explains a real‑world migration from MySQL to ClickHouse that reduced a 3‑minute query to under one second, details installation, migration methods, performance results, synchronization options, and common pitfalls.
This article details a production incident where Redis memory usage surged due to improper Set serialization, explains the investigation of storage structures and capacity formulas, and outlines a step‑by‑step data migration and cleanup process to restore normal operation.
This article introduces ClickHouse, a column‑store OLAP database, explains its advantages over row‑store systems, guides installation on macOS, details migration strategies from MySQL, presents performance benchmarks showing 200‑fold speedups, and discusses synchronization methods, pitfalls, and practical tips.
This article introduces ClickHouse as a high‑performance columnar OLAP database, compares it with row‑based systems, details migration methods from MySQL, presents performance benchmarks showing dramatic speed gains, discusses data synchronization strategies, explains why ClickHouse is fast, and shares common pitfalls and solutions.
The article explains why MySQL databases need regular backup, describes common data‑loss scenarios, classifies backup types (hot, cold, warm) and methods (logical, physical), lists popular tools, and provides detailed command‑line examples for backing up, restoring, migrating, and exporting data.
This article presents a comprehensive case study of migrating a 200‑plus node Hadoop offline platform across data centers, covering background, architecture, solution evaluation, detailed implementation steps, consistency checks, operational safeguards, encountered issues, and future recommendations.
Alibaba migrated Xianyu Coin’s points from the legacy KingTower platform to the new Banliang service using a four‑phase, service‑based approach—preparation, active/passive data migration with distributed locks, dual‑write for consistency, reconciliation, and a gradual service switch—completing the transition in a month without user impact.
This article presents a comprehensive case study of 58.com’s massive Hadoop‑based offline computing platform, detailing its architecture, scaling challenges, performance‑tuning measures, YARN and SparkSQL upgrades, and the systematic cross‑data‑center migration of thousands of nodes and petabytes of data.
