BestHub
Sign in
Tagged articles

Data Migration

259 articles · Page 2 of 3
Open Source Linux
Open Source Linux
Jul 4, 2023 · Operations

Master Redis Monitoring, Migration, and Cluster Management with Prometheus and CacheCloud

This guide walks through essential Redis operations, covering real‑time monitoring with the INFO command and Prometheus‑compatible exporters, data migration using Redis‑shake, consistency verification via Redis‑full‑check, and comprehensive cluster management with CacheCloud, providing practical tools for reliable Redis administration.

Data MigrationMonitoringOperations
0 likes · 11 min read
Master Redis Monitoring, Migration, and Cluster Management with Prometheus and CacheCloud
Architect's Guide
Architect's Guide
Jun 15, 2023 · Databases

Splitting a Massive MySQL Financial Transaction Table: Challenges, Design, and Implementation

The article describes how a finance team tackled a 50‑million‑row MySQL transaction table by analyzing the pain points, defining split goals, selecting sharding‑jdbc, designing multi‑source transaction handling, pagination across shards, data migration strategies, and a staged rollout to ensure stability and performance.

Data MigrationDatabase ScalingMySQL
0 likes · 10 min read
Splitting a Massive MySQL Financial Transaction Table: Challenges, Design, and Implementation
dbaplus Community
dbaplus Community
Jun 11, 2023 · Databases

Splitting a 500‑Million‑Row MySQL Table: Practical Sharding‑JDBC Lessons

Facing a MySQL transaction table exceeding 50 million rows and growing rapidly, a finance team redesigned the schema using sharding‑jdbc, chose transaction time as the shard key, tackled multi‑data‑source transactions and cross‑shard pagination, and executed a phased migration to ensure stability and performance.

Data MigrationLarge TablesMySQL
0 likes · 11 min read
Splitting a 500‑Million‑Row MySQL Table: Practical Sharding‑JDBC Lessons
Top Architect
Top Architect
May 29, 2023 · Databases

Large Table Splitting in MySQL: Challenges, Solutions, and Lessons Learned

Facing a 50‑million‑row financial transaction table, the team implemented horizontal sharding with sharding‑jdbc, designed migration and pagination strategies, tackled multi‑datasource transaction issues, and outlined a phased rollout, providing practical insights for large‑scale MySQL table splitting and backend system stability.

Data MigrationMySQLSharding
0 likes · 11 min read
Large Table Splitting in MySQL: Challenges, Solutions, and Lessons Learned
Architecture Digest
Architecture Digest
May 16, 2023 · Backend Development

High‑Availability Architecture for a Membership System: Dual‑Center ES Cluster, Redis Caching, MySQL Migration, and Fine‑Grained Flow Control

This article presents a comprehensive engineering case study of a high‑traffic membership system, detailing the dual‑center Elasticsearch high‑availability design, traffic‑isolated three‑cluster ES architecture, Redis caching strategy, dual‑center MySQL partitioning and migration plan, abnormal member relationship governance, and future fine‑grained flow‑control and downgrade policies.

Backend ArchitectureData MigrationElasticsearch
0 likes · 19 min read
High‑Availability Architecture for a Membership System: Dual‑Center ES Cluster, Redis Caching, MySQL Migration, and Fine‑Grained Flow Control
Architect
Architect
May 13, 2023 · Databases

Splitting a Massive MySQL Financial Transaction Table: Challenges, Strategies, and Implementation

The article details how a finance team tackled a 50‑million‑row MySQL transaction table by analyzing the pre‑split problems, defining split goals, choosing sharding‑jdbc, addressing multi‑datasource transaction and pagination issues, designing a phased migration and rollout plan, and summarizing lessons learned.

Data MigrationMySQLSharding
0 likes · 13 min read
Splitting a Massive MySQL Financial Transaction Table: Challenges, Strategies, and Implementation
Efficient Ops
Efficient Ops
May 9, 2023 · Databases

Building a Robust Database Consistency Check Tool for Seamless Migrations

This article examines the challenges of data consistency verification during database migrations, reviews existing tools like pt-table-checksum, sync‑diff‑inspector and gt‑checksum, outlines essential capabilities, and proposes a detailed workflow and future enhancements for a versatile, low‑impact consistency checking solution.

Data MigrationMySQLchecksum tools
0 likes · 11 min read
Building a Robust Database Consistency Check Tool for Seamless Migrations
ITPUB
ITPUB
Apr 25, 2023 · Big Data

Top 8 Open‑Source ETL Tools for Data Migration and Integration

This article reviews eight widely used ETL and data‑migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their core features, architectures, supported data sources, and typical usage scenarios to help practitioners choose the right solution.

Big DataData IntegrationData Migration
0 likes · 13 min read
Top 8 Open‑Source ETL Tools for Data Migration and Integration
dbaplus Community
dbaplus Community
Apr 19, 2023 · Databases

How to Migrate Large Tables and Sync Data Seamlessly in Production

This article explains common business scenarios such as large‑table splitting, cross‑database migration, and data synchronization, then compares loss‑bearing and loss‑less migration strategies, and provides detailed step‑by‑step implementation guidance for smooth dual‑write migration and real‑world tooling.

Data MigrationDatabasesSharding
0 likes · 17 min read
How to Migrate Large Tables and Sync Data Seamlessly in Production
macrozheng
macrozheng
Mar 27, 2023 · Big Data

Top 8 Open-Source ETL Tools for Efficient Data Migration

This guide reviews eight popular ETL and data migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their core features, architectures, and use cases to help engineers choose the right solution for reliable data integration.

Big DataData IntegrationData Migration
0 likes · 14 min read
Top 8 Open-Source ETL Tools for Efficient Data Migration
Su San Talks Tech
Su San Talks Tech
Mar 24, 2023 · Big Data

Top 8 Open-Source ETL Tools You Should Know for Efficient Data Migration

Explore a comprehensive overview of eight popular ETL and data migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their features, architectures, and use cases to help you choose the right solution for efficient data integration.

Big DataData IntegrationData Migration
0 likes · 13 min read
Top 8 Open-Source ETL Tools You Should Know for Efficient Data Migration
DataFunSummit
DataFunSummit
Feb 27, 2023 · Big Data

Gaotu Data Platform Cloud Migration and Cost Management Case Study

This article presents a detailed case study of Gaotu's migration of its on‑premises big‑data platform to Tencent Cloud, covering background challenges, a phased migration plan, implementation lessons, cost‑control measures, and future optimization strategies, with practical insights for similar enterprises.

Cloud ComputingData MigrationGaotu
0 likes · 22 min read
Gaotu Data Platform Cloud Migration and Cost Management Case Study
Liangxu Linux
Liangxu Linux
Jan 2, 2023 · Databases

13 Must‑Know MySQL Functions and Tricks for Everyday Development

This guide walks through a collection of practical MySQL functions and commands—including GROUP_CONCAT, CHAR_LENGTH, LOCATE, REPLACE, NOW, INSERT…SELECT, INSERT…IGNORE, SELECT FOR UPDATE, ON DUPLICATE KEY UPDATE, SHOW CREATE TABLE, CREATE TABLE … SELECT, EXPLAIN, SHOW PROCESSLIST and mysqldump—providing clear syntax, example queries, result screenshots and key usage notes to help developers write more efficient and reliable SQL.

Data MigrationDatabaseMySQL
0 likes · 14 min read
13 Must‑Know MySQL Functions and Tricks for Everyday Development
ITPUB
ITPUB
Jan 2, 2023 · Databases

Mastering MySQL Sharding: When and How to Split Databases and Tables

Facing massive user and order data growth, this article examines storage options, compares relational, NoSQL, and NewSQL databases, and provides a detailed guide to implementing MySQL‑based sharding with proxy and client modes, key selection, partition strategies, migration steps, and future scaling considerations.

Data MigrationMySQLdatabase partitioning
0 likes · 12 min read
Mastering MySQL Sharding: When and How to Split Databases and Tables
JD Tech
JD Tech
Dec 7, 2022 · Databases

Practical Guide to Large‑Scale Data Migration Using Sharding‑Proxy

This article presents a step‑by‑step practice of migrating massive billing data to 32 sharded databases with Sharding‑Proxy, covering background, objectives, four migration solutions, detailed proxy installation and configuration, debugging, migration workflow, data validation, common issues and their resolutions.

ConfigurationData MigrationDatabase
0 likes · 14 min read
Practical Guide to Large‑Scale Data Migration Using Sharding‑Proxy
Programmer DD
Programmer DD
Nov 21, 2022 · Databases

How to Achieve Seamless Horizontal Scaling with 2N Expansion and Keepalived

This guide explains how to scale a sharded database horizontally by adding new nodes, handling data migration with stop‑service, stop‑write, log‑based, dual‑write, and smooth 2N strategies, and implementing high‑availability using MariaDB double‑master replication and Keepalived, complete with configuration examples and code snippets.

Data MigrationHigh AvailabilityKeepalived
0 likes · 35 min read
How to Achieve Seamless Horizontal Scaling with 2N Expansion and Keepalived
dbaplus Community
dbaplus Community
Oct 27, 2022 · Databases

How to Safely Delete Hundreds of Millions of Rows Without Locking Your Database

This guide explains why a single massive DELETE on a 500‑million‑row table creates long‑running transactions and locks, and shows step‑by‑step techniques—date‑range batching, primary‑key range slicing, and insert‑instead‑of‑delete—to break the operation into manageable chunks and improve performance.

Data MigrationLarge TableSQL
0 likes · 8 min read
How to Safely Delete Hundreds of Millions of Rows Without Locking Your Database
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Sep 26, 2022 · Databases

Data Migration & Distributed Transactions: XA, BASE, TCC, AT Demystified

This article examines how vertical and horizontal data partitioning leads to migration and consistency challenges, outlines full, full‑plus‑incremental, and binlog‑based migration strategies, and compares strong‑consistency XA transactions with flexible BASE approaches such as TCC and AT, including their components, workflows, advantages, and drawbacks.

BASEData MigrationTCC
0 likes · 9 min read
Data Migration & Distributed Transactions: XA, BASE, TCC, AT Demystified
FunTester
FunTester
Sep 25, 2022 · Databases

Data Migration Scenarios, Testing, and Acceptance Guidelines

This article outlines common data migration scenarios such as system consolidation and database sharding, details analysis of user data before migration, discusses conflict resolution rules, presents migration planning and acceptance testing steps, and highlights post‑release monitoring and user feedback handling.

Data MigrationOperationsacceptance
0 likes · 7 min read
Data Migration Scenarios, Testing, and Acceptance Guidelines
Code Ape Tech Column
Code Ape Tech Column
Sep 24, 2022 · Operations

Overview of Redis Monitoring, Data Migration, and Cluster Management Tools

This article introduces essential Redis operational tools, covering real‑time monitoring with the INFO command and Prometheus‑exporter, data migration using Redis‑shake, consistency checking via Redis‑full‑check, and cluster management through CacheCloud, providing practical guidance for administrators.

Cluster ManagementData MigrationOperations
0 likes · 10 min read
Overview of Redis Monitoring, Data Migration, and Cluster Management Tools
Tencent Cloud Developer
Tencent Cloud Developer
Sep 13, 2022 · Databases

Implementing a Transparent Data Encryption Framework for Database Security

The article presents a transparent data‑encryption framework for database security that uses an AOP‑based driver interceptor to rewrite SQL, encrypt/decrypt sensitive fields, manage performance with locks and caching, support configurable algorithms via SPI/YAML, and guide a three‑phase migration with minimal code intrusion.

AOPData MigrationDatabase Security
0 likes · 11 min read
Implementing a Transparent Data Encryption Framework for Database Security
Baidu Geek Talk
Baidu Geek Talk
Aug 17, 2022 · Industry Insights

How Baidu Cloud Storage Solves the Four Big Challenges of the ABC Era

This article examines the massive data, cost, stability, and diversity challenges of the AI‑driven, big‑data, cloud‑first "ABC" era and explains how Baidu's Canghai storage portfolio—including BOS, CDS, CFS, PFS, RapidFS, CloudFlow, and storage gateways—addresses each issue through scalable architecture, tiered lifecycle policies, multi‑AZ disaster recovery, and integrated hybrid‑cloud solutions.

BaiduCloud StorageData Migration
0 likes · 16 min read
How Baidu Cloud Storage Solves the Four Big Challenges of the ABC Era
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Aug 15, 2022 · Cloud Computing

How Baidu’s Canghai Storage Tackles Massive Data Challenges in the Cloud

This article outlines the four major storage challenges of the ABC era—massive scale, cost efficiency, stability, and diversity—and explains how Baidu’s Canghai storage suite, including BOS, CDS, CFS, PFS, RapidFS, CloudFlow, and storage gateways, addresses each through multi‑cloud migration, tiered lifecycle management, and robust disaster‑recovery solutions.

AIBig DataCloud Storage
0 likes · 15 min read
How Baidu’s Canghai Storage Tackles Massive Data Challenges in the Cloud
Aikesheng Open Source Community
Aikesheng Open Source Community
Aug 15, 2022 · Databases

Step-by-Step Migration of Large MySQL Datasets to DBLE Using TiDB DM

This article provides a comprehensive, code‑rich guide on using TiDB DM v2.0.7 to perform full and incremental migration of terabyte‑scale MySQL data to DBLE, covering environment setup, TiDB DM installation, DBLE configuration, data generation, task execution, verification, and important operational considerations.

DBLEData MigrationDatabase Replication
0 likes · 18 min read
Step-by-Step Migration of Large MySQL Datasets to DBLE Using TiDB DM
DaTaobao Tech
DaTaobao Tech
Jul 12, 2022 · Backend Development

Stable System Architecture Upgrade: Interface and Data Migration Strategy

The article outlines a proxy‑based migration framework that enables stable system architecture upgrades by using total and gray‑scale switches to achieve gradual rollout, rollback, non‑intrusive business logic, and graceful shutdown, while handling single‑query routing, multi‑query deduplication, double‑write updates, and direct inserts, illustrated with AspectJ‑driven code and retail store‑ID examples.

Data Migrationgray-scale deploymentinterface migration
0 likes · 11 min read
Stable System Architecture Upgrade: Interface and Data Migration Strategy
Big Data Technology Architecture
Big Data Technology Architecture
Jun 8, 2022 · Big Data

Bilibili Offline Computing Platform: Migration from Hive to Spark and Comprehensive Performance Optimizations

The article details Bilibili's evolution of its offline computing platform from Hadoop‑based Hive to Spark, describing migration tools, SQL conversion, result and resource comparison, shuffle stability, small‑file handling, runtime filters, data skipping, ZSTD support, Hive Metastore federation, traffic control, and future optimization directions.

Data MigrationHiveSpark
0 likes · 29 min read
Bilibili Offline Computing Platform: Migration from Hive to Spark and Comprehensive Performance Optimizations
Architecture Digest
Architecture Digest
Apr 4, 2022 · Databases

Splitting a Massive MySQL Financial Transaction Table: Challenges, Sharding Strategy, and Migration Process

This article describes how a finance team tackled a 500 million‑row MySQL transaction table by analyzing pre‑split issues, defining sharding goals, selecting sharding‑jdbc, addressing multi‑datasource transaction and pagination challenges, designing a hybrid data‑migration plan, and executing a three‑stage rollout to ensure system stability and performance.

Data MigrationDatabase ScalingMySQL
0 likes · 11 min read
Splitting a Massive MySQL Financial Transaction Table: Challenges, Sharding Strategy, and Migration Process
vivo Internet Technology
vivo Internet Technology
Mar 30, 2022 · Backend Development

Design of a Bloom Filter‑Based Video Recommendation Deduplication Service for Short Video Platforms

The paper proposes a Bloom‑filter‑based deduplication service for short‑video recommendation that moves three‑month playback histories to disk‑backed Bloom filters while keeping the latest 100 served IDs in Redis, employing write batching, sharding, expiration policies, and an incremental migration strategy to replace memory‑intensive Redis ZSets and dramatically reduce storage costs.

Data Migrationdeduplicationdisk KV
0 likes · 21 min read
Design of a Bloom Filter‑Based Video Recommendation Deduplication Service for Short Video Platforms
dbaplus Community
dbaplus Community
Mar 7, 2022 · Databases

How to Prevent Null, Type, and Charset Pitfalls in Oracle‑to‑ADB Data Sync

This article details the common pitfalls encountered when synchronizing Oracle databases to AnalyticDB PostgreSQL, covering null versus empty string handling, data type conversion challenges, character set issues, special character processing, and comprehensive testing strategies to ensure data consistency and performance.

AnalyticDBData MigrationNULL handling
0 likes · 17 min read
How to Prevent Null, Type, and Charset Pitfalls in Oracle‑to‑ADB Data Sync
Architect
Architect
Feb 18, 2022 · Cloud Native

Large‑Scale etcd Cluster Performance Optimization and Pod Data Splitting in Ant Group’s Sigma

This article describes how Ant Group tackled the performance ceiling of its massive Sigma Kubernetes clusters by horizontally splitting etcd storage for Pods, Leases and Events, redesigning watch handling to avoid component restarts, and using snapshot‑based migration to preserve data integrity while reducing latency.

Cluster PerformanceData MigrationKubernetes
0 likes · 27 min read
Large‑Scale etcd Cluster Performance Optimization and Pod Data Splitting in Ant Group’s Sigma
转转QA
转转QA
Feb 17, 2022 · Backend Development

Building a Generic API Diff Tool for System Refactoring and Data Migration Testing

This article describes how a Python‑based diff utility was created to compare old and new API responses during a promotion change‑record migration, detailing its implementation, handling of JSON structures, improvements over the first version, and plans for integration with an API testing platform.

API testingData MigrationPython
0 likes · 8 min read
Building a Generic API Diff Tool for System Refactoring and Data Migration Testing
IT Xianyu
IT Xianyu
Jan 14, 2022 · Operations

Redis Monitoring, Data Migration, and Cluster Management Tools Overview

This article introduces essential Redis operational tools, covering the INFO command for monitoring, Prometheus‑based redis‑exporter visualization, the Redis‑shake data migration utility, Redis‑full‑check consistency verification, and the CacheCloud platform for comprehensive cluster management.

CacheCloudData MigrationMonitoring
0 likes · 10 min read
Redis Monitoring, Data Migration, and Cluster Management Tools Overview
Code Ape Tech Column
Code Ape Tech Column
Dec 23, 2021 · Operations

Overview of Redis Monitoring, Data Migration, and Cluster Management Tools

This article introduces essential Redis operational tools, covering the INFO command for real‑time monitoring, Prometheus‑based exporters, data migration utilities like Redis‑shake, consistency checking with Redis‑full‑check, and cluster management via CacheCloud, while offering practical usage tips.

CacheCloudData MigrationRedis
0 likes · 10 min read
Overview of Redis Monitoring, Data Migration, and Cluster Management Tools
dbaplus Community
dbaplus Community
Dec 15, 2021 · Big Data

How We Migrated Hundreds of Petabytes of Hadoop Data Without Downtime

This article details the background, challenges, and step‑by‑step solutions for migrating over a hundred petabytes of Hadoop HDFS data across data centers within a month, covering strategy selection, code modifications, balance optimization, and tool enhancements.

Balance OptimizationBig Data OperationsData Migration
0 likes · 14 min read
How We Migrated Hundreds of Petabytes of Hadoop Data Without Downtime
Top Architect
Top Architect
Dec 11, 2021 · Databases

Scaling Zhihu’s Moneta Service with TiDB: Architecture, Performance, and Lessons Learned

Zhihu’s Moneta service, handling over a trillion rows and billions of daily writes, migrated from MySQL to TiDB, achieving millisecond query latency, high availability, and horizontal scalability, and the article details the architecture, performance metrics, migration challenges, and lessons learned from this large‑scale deployment.

Data MigrationTiDBdatabase scalability
0 likes · 13 min read
Scaling Zhihu’s Moneta Service with TiDB: Architecture, Performance, and Lessons Learned
DevOps
DevOps
Oct 20, 2021 · Backend Development

Microservice Architecture: Stability, Service Degradation, Data Consistency, and Migration Practices

This article summarizes the author's extensive experience with microservice adoption, covering its benefits, the challenges of stability, service degradation strategies, distributed transaction patterns, data migration techniques, and practical monitoring using APM tools to help teams successfully transform to microservices.

APMData MigrationMicroservices
0 likes · 35 min read
Microservice Architecture: Stability, Service Degradation, Data Consistency, and Migration Practices
Selected Java Interview Questions
Selected Java Interview Questions
Aug 25, 2021 · Databases

ClickHouse Overview: Architecture, MySQL Migration, Performance Testing, and Practical Tips

This article introduces ClickHouse, a high‑performance open‑source columnar database, explains its architecture versus row‑based systems, details migration from MySQL, showcases installation, performance benchmarks, data‑sync strategies, common pitfalls, and summarizes its benefits for large‑scale analytical workloads.

Big DataClickHouseColumnar Database
0 likes · 7 min read
ClickHouse Overview: Architecture, MySQL Migration, Performance Testing, and Practical Tips
Programmer DD
Programmer DD
Aug 22, 2021 · Databases

How ClickHouse Boosted Query Speed 200×: A Practical Migration Guide

This article introduces ClickHouse as a column‑oriented OLAP database, compares it with traditional row‑based databases, and details a real‑world migration from MySQL that reduced query time from minutes to under one second, along with performance testing, sync strategies, and common pitfalls.

ClickHouseColumnar DatabaseData Migration
0 likes · 7 min read
How ClickHouse Boosted Query Speed 200×: A Practical Migration Guide
Tencent Cloud Middleware
Tencent Cloud Middleware
Aug 19, 2021 · Backend Development

Fast Kafka Cluster Expansion: Practical Strategies to Reduce Data Migration

When a Kafka cluster reaches load limits or experiences sudden traffic spikes, urgent expansion is needed, but data migration can be time‑consuming and risky; this guide outlines several practical techniques—including adjusting retention, adding partitions, leader switching, and single‑replica operation—to quickly scale clusters while minimizing data movement and service disruption.

Data MigrationKafkaPartition Reassignment
0 likes · 21 min read
Fast Kafka Cluster Expansion: Practical Strategies to Reduce Data Migration
Maoyan Technology Team
Maoyan Technology Team
Jul 8, 2021 · Backend Development

How Maoyan Unified Billions of App and Mini‑Program Accounts

This article details Maoyan's multi‑account integration journey, describing the background, challenges, industry research, and a three‑phase write‑through solution that merges APP and mini‑program accounts using phone numbers as a common anchor while ensuring data consistency, scalability, and minimal user impact.

Data Migrationaccount-integrationscalability
0 likes · 18 min read
How Maoyan Unified Billions of App and Mini‑Program Accounts
ZhiKe AI
ZhiKe AI
Jul 2, 2021 · Databases

Fast Migration of Millions of MySQL Rows: Step-by-Step Guide

This article outlines a practical method for migrating MySQL tables with millions of rows by exporting data to an outfile, compressing, transferring, and loading it on the target server, while also addressing strict SQL mode errors that can arise during import.

Data MigrationLOAD DATALarge Tables
0 likes · 2 min read
Fast Migration of Millions of MySQL Rows: Step-by-Step Guide
21CTO
21CTO
Apr 1, 2021 · Backend Development

When and How to Refactor Legacy Systems into Microservices

This article explains the signs that indicate a legacy system needs refactoring, describes the challenges of a monolithic architecture with massive tables, outlines goals such as zero business impact and migration to microservices, and details a step‑by‑step migration strategy using data pipelines and careful interface compatibility.

Data MigrationMicroservicesarchitecture
0 likes · 8 min read
When and How to Refactor Legacy Systems into Microservices
360 Smart Cloud
360 Smart Cloud
Apr 1, 2021 · Databases

Using TiDB Data Migration (DM) for MySQL‑to‑TiDB Sync: Architecture, Features, Tuning and Troubleshooting

This article shares practical experience with TiDB Data Migration (DM), covering its background, architecture, key features, online DDL support, common error handling such as duplicate‑key issues, large‑scale import tuning, configuration limits, and cleanup recommendations for reliable MySQL‑to‑TiDB synchronization.

DMData MigrationDatabase Synchronization
0 likes · 16 min read
Using TiDB Data Migration (DM) for MySQL‑to‑TiDB Sync: Architecture, Features, Tuning and Troubleshooting
Architect's Tech Stack
Architect's Tech Stack
Mar 27, 2021 · Databases

Introducing ClickHouse: Columnar Database Overview, MySQL Migration, Performance Comparison, and Practical Tips

This article introduces ClickHouse, explains its column‑oriented architecture versus row‑oriented databases, details a MySQL‑to‑ClickHouse migration, shows a performance benchmark that reduces query time from minutes to seconds, and shares practical deployment tips and common pitfalls.

ClickHouseColumnar DatabaseData Migration
0 likes · 8 min read
Introducing ClickHouse: Columnar Database Overview, MySQL Migration, Performance Comparison, and Practical Tips
Java Backend Technology
Java Backend Technology
Mar 11, 2021 · Databases

Combine Hash and Range Sharding to Prevent Hotspots Without Data Migration

This article explains how to split large-scale database tables using vertical and horizontal partitioning, compares hash-modulo and range-based sharding, and proposes a hybrid approach that leverages both methods to balance data distribution, avoid hotspot issues, and eliminate the need for costly data migrations during scaling.

Data MigrationHash ShardingRange Sharding
0 likes · 13 min read
Combine Hash and Range Sharding to Prevent Hotspots Without Data Migration
Code Ape Tech Column
Code Ape Tech Column
Mar 3, 2021 · Databases

When INSERT INTO SELECT Breaks MySQL: A Cautionary Data‑Migration Story

An engineer’s costly mistake using MySQL’s INSERT INTO SELECT for nightly data migration led to out‑of‑memory crashes, full‑table scans, and payment record loss, prompting a deep dive into locking behavior, transaction isolation, and how proper indexing can safely rescue large‑scale inserts.

Data MigrationINSERT INTO SELECTIndexing
0 likes · 7 min read
When INSERT INTO SELECT Breaks MySQL: A Cautionary Data‑Migration Story
Alibaba Cloud Developer
Alibaba Cloud Developer
Mar 1, 2021 · Databases

How We Scaled a High‑Volume Order System: Sharding, Migration, and Zero‑Downtime Strategies

This article details the end‑to‑end process of expanding a high‑traffic ride‑hailing order system on Alibaba Cloud, covering capacity planning, database sharding from 256 to 4096 tables, custom data‑sync middleware, validation, repair, and a gray‑release migration that achieved zero incidents.

Data MigrationDatabasesSharding
0 likes · 17 min read
How We Scaled a High‑Volume Order System: Sharding, Migration, and Zero‑Downtime Strategies
Architects' Tech Alliance
Architects' Tech Alliance
Feb 28, 2021 · Operations

Ensuring Data Integrity in Multi‑Cloud Migration: MySQL, File & Object Storage Strategies

This article outlines practical techniques for multi‑cloud data migration, covering challenges, resource categories, migration phases, and detailed synchronization and verification methods for MySQL databases, file systems, and object storage using tools such as UDTS, pt‑table‑checksum, sync_diff_inspector and US3SYNC.

Data MigrationData verificationMulti-Cloud
0 likes · 17 min read
Ensuring Data Integrity in Multi‑Cloud Migration: MySQL, File & Object Storage Strategies
JD Cloud Developers
JD Cloud Developers
Feb 25, 2021 · Operations

Boost Multi-Cloud Redis Migration with RedisSyncer

This article explains how RedisSyncer enables fast, reliable cross‑version and cross‑cluster Redis data migration in multi‑cloud environments, addressing challenges such as version mismatches, node inconsistencies, and rapid rollback while improving access speed and reducing database load.

Data MigrationDockerMulti-Cloud
0 likes · 5 min read
Boost Multi-Cloud Redis Migration with RedisSyncer
UCloud Tech
UCloud Tech
Feb 8, 2021 · Operations

Mastering Multi-Cloud Data Migration: MySQL, File & Object Storage Strategies

This article explains the challenges of cross‑cloud migration, outlines three resource categories, details MySQL, file and object storage synchronization methods, presents practical tools and validation techniques, and shares real‑world case studies to ensure data integrity and efficiency.

Data MigrationMulti-CloudMySQL
0 likes · 17 min read
Mastering Multi-Cloud Data Migration: MySQL, File & Object Storage Strategies
Sohu Tech Products
Sohu Tech Products
Feb 3, 2021 · Mobile Development

Understanding and Adapting Scoped Storage in Android 10+

This article explains Android's scoped storage model introduced in Android 10, details how it changes file access, provides migration strategies for new and legacy data, and includes practical code examples for handling media files, legacy storage flags, and common file‑operation errors across Android versions.

Data MigrationFile APILegacy Storage
0 likes · 22 min read
Understanding and Adapting Scoped Storage in Android 10+
Top Architect
Top Architect
Jan 18, 2021 · Big Data

Migrating Over 2 Billion MySQL Records to Google BigQuery Using Kafka

This article details a real‑world solution for migrating more than two billion MySQL records to Google BigQuery by streaming data through Kafka, employing partitioned tables, data filtering, and incremental migration to avoid downtime and reduce storage costs.

Big DataBigQueryData Migration
0 likes · 7 min read
Migrating Over 2 Billion MySQL Records to Google BigQuery Using Kafka
Top Architect
Top Architect
Jan 13, 2021 · Big Data

Migrating Over 2 Billion MySQL Records to the Cloud with Kafka and BigQuery

Facing a MySQL table with over 2 billion continuously growing rows, the team designed a cloud‑based solution using Kafka to stream data into Google BigQuery, applied partitioned schemas and data cleaning to improve query performance, reduce storage costs, and avoid downtime.

BigQueryData MigrationPartitioning
0 likes · 8 min read
Migrating Over 2 Billion MySQL Records to the Cloud with Kafka and BigQuery
DeWu Technology
DeWu Technology
Dec 18, 2020 · Databases

TiDB Architecture, Design, and Real‑World Application

TiDB is a distributed NewSQL database that combines MySQL compatibility with a stateless SQL layer, a PD metadata manager, and a transactional key‑value store, offering automatic scaling, multi‑data‑center Raft replication, global secondary indexes, Percolator‑style 2PC, HTAP via TiFlash, and a rich ecosystem of migration, backup, CDC, and monitoring tools, as demonstrated by its large‑scale deployment at DeWu.

Data MigrationDatabase ArchitectureHTAP
0 likes · 18 min read
TiDB Architecture, Design, and Real‑World Application
政采云技术
政采云技术
Dec 8, 2020 · Backend Development

Setting Up an NPM Private Registry: Docker Deployment, Data Migration, and OSS Disaster Recovery

This article provides a step‑by‑step guide to building an NPM private registry using Cnpmjs.org, covering containerized deployment with Docker, migrating packages from Verdaccio, implementing OSS disaster‑recovery backups, configuring email notifications, and tips for further customization.

Data MigrationPrivate Registrybackend
0 likes · 13 min read
Setting Up an NPM Private Registry: Docker Deployment, Data Migration, and OSS Disaster Recovery
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 7, 2020 · Databases

MySQL Billion‑Row Data Migration: Export, Import, and Performance Optimization

This article documents a step‑by‑step MySQL data migration case involving 140 million rows, covering export via SELECT INTO OUTFILE, import with LOAD DATA INFILE, and a series of performance tweaks such as index removal, session variable tuning, engine selection, and monitoring to achieve sub‑10‑minute load times.

Data MigrationExportMySQL
0 likes · 9 min read
MySQL Billion‑Row Data Migration: Export, Import, and Performance Optimization
360 Tech Engineering
360 Tech Engineering
Dec 4, 2020 · Databases

Practical Experience and Tips with TiDB Data Migration (DM) Tool

This article shares a comprehensive overview of TiDB Data Migration (DM), covering its architecture, configuration, online DDL support, common pitfalls such as duplicate‑key errors, large‑scale import tuning, version limits, and cleanup recommendations to help DBAs efficiently migrate MySQL/MariaDB workloads to TiDB.

DMData MigrationMySQL
0 likes · 16 min read
Practical Experience and Tips with TiDB Data Migration (DM) Tool
JD Cloud Developers
JD Cloud Developers
Dec 4, 2020 · Databases

How TiDB Solves High‑Traffic Database Challenges: Architecture, Features, and Real‑World Cases

This article summarizes a JD Cloud‑TiDB live webcast, explaining why JD Cloud chose TiDB, detailing its distributed architecture (TiDB Server, TiKV storage, PD), highlighting features such as horizontal scaling, strong consistency, real‑time analytics, backup, migration, and showcasing multiple enterprise migration case studies that demonstrate TiDB’s ability to overcome traditional database bottlenecks.

Cloud DatabaseData MigrationReal-Time Analytics
0 likes · 13 min read
How TiDB Solves High‑Traffic Database Challenges: Architecture, Features, and Real‑World Cases
FunTester
FunTester
Dec 4, 2020 · Operations

Mastering Data Migration Testing: A Complete 3‑Phase Guide

This guide outlines the three‑stage data‑migration testing process—preparation, execution, and verification—detailing how to assess data scope, choose migration methods, evaluate system compatibility, construct test scenarios, validate scripts, monitor performance, and confirm backend, frontend, and business‑logic integrity after migration.

Data MigrationPerformance Testingbackend verification
0 likes · 7 min read
Mastering Data Migration Testing: A Complete 3‑Phase Guide
vivo Internet Technology
vivo Internet Technology
Nov 25, 2020 · Backend Development

How Vivo Leverages Alibaba Canal for Zero‑Downtime Data Migration and HA

This article explains how Vivo uses Alibaba's open‑source Canal to capture MySQL binlog changes, achieve non‑stop sharding and cross‑region data migrations, ensure high availability with Zookeeper, and share practical lessons on serialization, consistency, and monitoring in large‑scale backend systems.

CanalData MigrationHigh Availability
0 likes · 16 min read
How Vivo Leverages Alibaba Canal for Zero‑Downtime Data Migration and HA
Tencent Tech
Tencent Tech
Nov 19, 2020 · Cloud Computing

How Tencent Built a Massive Cloud Storage System to Power QQ Album and Beyond

This article chronicles Tencent's journey from the early development of the TFS distributed storage platform to large‑scale data migrations, flexible bandwidth strategies, and the creation of the cloud‑native YottaStore, illustrating how a small architecture team solved massive storage challenges for billions of users.

Big DataCloud StorageData Migration
0 likes · 15 min read
How Tencent Built a Massive Cloud Storage System to Power QQ Album and Beyond
Xianyu Technology
Xianyu Technology
Nov 19, 2020 · Operations

Rapid and Safe Migration of a Centralized Microservice Platform to Department‑Built Infrastructure

The team migrated a large, multi‑service microservice publishing platform—including Xianyu, Taobao, Alipay, and Tmall—from a centralized environment to a department‑built infrastructure in ten working days by cloning the repo, updating configurations, separating databases, rigorously verifying functionality across dev, pre‑release, and production, and ensuring isolation and monitoring for stability.

Data MigrationOperationsbackend development
0 likes · 7 min read
Rapid and Safe Migration of a Centralized Microservice Platform to Department‑Built Infrastructure
21CTO
21CTO
Nov 9, 2020 · Databases

How ClickHouse Turns MySQL Bottlenecks into Sub‑Second OLAP Queries

This article introduces ClickHouse, compares column‑store and row‑store databases, shows how migrating a 50‑million‑row MySQL table to ClickHouse reduced query time from minutes to under one second, and shares practical installation, migration, performance testing, and synchronization tips.

ClickHouseColumnar DatabaseData Migration
0 likes · 6 min read
How ClickHouse Turns MySQL Bottlenecks into Sub‑Second OLAP Queries
ITPUB
ITPUB
Nov 5, 2020 · Databases

How ClickHouse Cut MySQL Query Time 200× – A Practical Migration Guide

This article introduces ClickHouse, compares column‑ and row‑oriented storage, explains a real‑world migration from MySQL to ClickHouse that reduced a 3‑minute query to under one second, details installation, migration methods, performance results, synchronization options, and common pitfalls.

ClickHouseColumnar DatabaseData Migration
0 likes · 7 min read
How ClickHouse Cut MySQL Query Time 200× – A Practical Migration Guide
Top Architect
Top Architect
Sep 25, 2020 · Databases

Introduction to ClickHouse: Installation, MySQL Migration, Performance Testing, and Best Practices

This article introduces ClickHouse, a column‑store OLAP database, explains its advantages over row‑store systems, guides installation on macOS, details migration strategies from MySQL, presents performance benchmarks showing 200‑fold speedups, and discusses synchronization methods, pitfalls, and practical tips.

ClickHouseColumnar DatabaseData Migration
0 likes · 6 min read
Introduction to ClickHouse: Installation, MySQL Migration, Performance Testing, and Best Practices
Architect
Architect
Sep 19, 2020 · Databases

ClickHouse Overview, MySQL Migration, Performance Testing, and Practical Tips

This article introduces ClickHouse as a high‑performance columnar OLAP database, compares it with row‑based systems, details migration methods from MySQL, presents performance benchmarks showing dramatic speed gains, discusses data synchronization strategies, explains why ClickHouse is fast, and shares common pitfalls and solutions.

ClickHouseColumnar DatabaseData Migration
0 likes · 5 min read
ClickHouse Overview, MySQL Migration, Performance Testing, and Practical Tips
Laravel Tech Community
Laravel Tech Community
Sep 5, 2020 · Databases

MySQL Database Backup and Recovery

The article explains why MySQL databases need regular backup, describes common data‑loss scenarios, classifies backup types (hot, cold, warm) and methods (logical, physical), lists popular tools, and provides detailed command‑line examples for backing up, restoring, migrating, and exporting data.

BackupData MigrationHot Backup
0 likes · 27 min read
MySQL Database Backup and Recovery
Xianyu Technology
Xianyu Technology
Jun 9, 2020 · Backend Development

Xianyu Coin Service Migration: A Service‑Based Data Migration Approach

Alibaba migrated Xianyu Coin’s points from the legacy KingTower platform to the new Banliang service using a four‑phase, service‑based approach—preparation, active/passive data migration with distributed locks, dual‑write for consistency, reconciliation, and a gradual service switch—completing the transition in a month without user impact.

Data MigrationReconciliationRedis Lock
0 likes · 11 min read
Xianyu Coin Service Migration: A Service‑Based Data Migration Approach
Big Data Technology Architecture
Big Data Technology Architecture
Jun 4, 2020 · Big Data

58.com Big Data Offline Computing Platform: Architecture, Scaling, Optimization, and Cross‑Data‑Center Migration

This article presents a comprehensive case study of 58.com’s massive Hadoop‑based offline computing platform, detailing its architecture, scaling challenges, performance‑tuning measures, YARN and SparkSQL upgrades, and the systematic cross‑data‑center migration of thousands of nodes and petabytes of data.

Big DataData MigrationHadoop
0 likes · 23 min read
58.com Big Data Offline Computing Platform: Architecture, Scaling, Optimization, and Cross‑Data‑Center Migration
dbaplus Community
dbaplus Community
Apr 12, 2020 · Databases

Why and How to Migrate from MongoDB to Elasticsearch: A Practical Guide

This article explains the motivations for moving a high‑volume operation‑log system from MongoDB to Elasticsearch, outlines the existing architecture, details capacity planning, index design, and a step‑by‑step migration process using Kafka, DataX, and Spring Boot, and shares the performance gains and lessons learned.

Data MigrationDataXDatabase Architecture
0 likes · 14 min read
Why and How to Migrate from MongoDB to Elasticsearch: A Practical Guide
Aikesheng Open Source Community
Aikesheng Open Source Community
Mar 30, 2020 · Databases

Getting Started with dtle: Installation, Configuration, and MySQL Data Migration Guide

This guide introduces dtle, an open‑source MySQL data‑transfer middleware, covering download, installation on three nodes, detailed dtle.conf configuration, startup commands, creating source and target users, preparing job.json, using HTTP API to create and monitor migration jobs, and verifying data migration results.

Data MigrationHTTP APIJob
0 likes · 11 min read
Getting Started with dtle: Installation, Configuration, and MySQL Data Migration Guide