The article details the migration of dbt metric definitions to Snowflake semantic views via Apache Ossie, revealing that rolling window metrics degrade to simple sums and cross-table filters cause import errors, requiring manual reconstruction and query-granularity validation to ensure numerical parity.
This article details Xiaohongshu's migration of its core real-time indexing pipeline from Kafka to Apache Fluss, covering architectural drivers like read amplification and historical replay contention, Fluss's columnar storage and Lakestream integration with Paimon, and a phased migration approach including baseline testing, shadow dual-run, and gradual cutover.
This article details the design and implementation of a custom database sharding component built on MyBatis plugin architecture, covering sharding strategies, read/write policies (double-write/double-read), SQL parsing and rewriting for COUNT/SUM/ORDER BY, routing algorithms with distributed locks, parameter mapping handling, and solutions to migration challenges like ORDER BY alias errors and aggregate query merging.
The article analyzes two online travel case studies: Tongcheng Travel migrating from Hudi to Paimon to solve slow large-table real-time writes using LSM-tree storage, and Trip.com adopting StarRocks with materialized views to unify lake queries and eliminate data warehouse copies, detailing migration methodologies, architectural reasoning, and production troubleshooting.
RustFS, a Rust-written S3-compatible distributed object storage, hits GA with 32.5K GitHub stars, offering MinIO migration paths including on-demand data migration, versioning, erasure coding, and storage pool scaling, plus a Docker quick-start for production evaluation.
The article explains why high‑traffic MySQL deployments hit performance limits, introduces the concepts of vertical and horizontal sharding, and provides a step‑by‑step guide—including necessity assessment, shard key selection, schema design, middleware integration, and data migration—using an e‑commerce order system as a concrete example.
At Apache COC Asia 2026, Didi’s engineers presented three deep‑dive sessions covering the evolution of massive‑scale object storage with Ozone, cost‑effective lakehouse migration using Iceberg and real‑time ingestion, and AI‑friendly enhancements to the open‑source Seata project, detailing architectural choices, performance gains, and open‑source contributions.
In this interview, senior DBA Yao Wei shares over a decade of hands‑on experience designing, testing, and operating heterogeneous database ecosystems for retail, internet, and financial services, detailing pre‑emptive DBA involvement, compatibility pitfalls, high‑availability strategies, automated validation pipelines, and migration to domestic databases.
Manus announced it will reverse its $2 billion acquisition by Meta, restore independent operations, and force all users to back up data before a tight migration window ending on August 23, while detailing its rapid growth, regulatory hurdles, and soaring ARR now estimated at $4‑5 billion.
When migrating tens of billions of rows, the article explains how a simple progress‑tracking table and failure‑log table enable automatic checkpoint‑based resumption, stateless execution, and dynamic batch tuning without restarting or rewriting code.
The article reviews SQLark, a Chinese one‑stop database development and management tool that supports Windows, macOS, and Linux, connects to major databases such as Oracle, MySQL and PostgreSQL, and integrates AI assistance, positioning it as a convenient, efficient alternative to Navicat and DBeaver.
iQIYI’s big‑data team built a self‑developed QBFS virtual file system that unifies private and multiple public clouds, providing transparent routing, automatic migration, intelligent caching and fine‑grained governance, which together reduce storage and compute costs by more than 30 % while supporting scalable analytics.
The author compares common database clients, highlights the risks of cracked Navicat and limited DBeaver, then details how SQLark’s cross‑platform support, AI‑driven features, smart SQL editor, data migration, ER diagram generation, and test‑data generator provide a more efficient and secure workflow for developers.
The article explains how changing enum values in a contract approval system without handling legacy data caused failures, outlines the time gap between code deployment and data flow, and provides a systematic compatibility strategy—including retaining old enum values, semantic checks, and a pre‑release checklist—to prevent similar incidents.
This article outlines a comprehensive, risk‑controlled SOP for migrating business data—including model changes, storage shifts, incremental dual‑write, back‑filling, full and incremental consistency checks, read‑switching, and final decommissioning—backed by concrete SQL examples and visual diagrams.
This article walks through three concrete methods for moving massive ClickHouse datasets—up to 100 billion rows—to Doris, detailing catalog integration, file export with stream load, and Spark‑based pipelines, while sharing real‑world performance results and pitfalls.
LoongCollector introduces a One‑Time file collection mode that scans matching files once, records a snapshot, and exits, enabling efficient historic log migration, data back‑fill, and temporary batch processing while providing fine‑grained checkpoints, execution windows, and throttling controls to avoid quota issues and ensure reliable completion.
In under a month the author migrated more than 100 heterogeneous tables from MySQL and MongoDB to PostgreSQL across isolated networks by automating Navicat's import/export logic, streaming data, handling special characters, configuring flexible sync strategies, and using OSS as a bridge to avoid OOM and lock‑contention.
Facing a ClickHouse cluster shutdown, the author explores three migration methods—using Doris’s ClickHouse catalog, exporting to files with Broker/Stream Load, and Spark—to transfer ~10 billion rows to Doris, evaluating each for simplicity, bugs, and performance, and sharing detailed steps, code snippets, and benchmark results.
This article outlines a practical sharding strategy for e‑commerce order systems, estimating future load, detailing user‑centric partitioning, heterogeneous designs for merchants and operators, and migration steps to achieve high‑concurrency writes and massive storage without downtime.
This article explains why interviewers probe sharding, walks through the typical chain of questions, compares partition tables and read/write splitting, shows how to decide between sharding strategies, estimate capacity, plan expansion, and safely migrate data with traffic replication.
This guide walks senior backend engineers through the strategic reasoning, capacity estimation, and step‑by‑step migration techniques for sharding databases, covering when to split, choosing between partitioning, read/write splitting, or full sharding, and how to plan safe expansions.
This article details the complete redesign of a fifteen‑year‑old e‑commerce review system, covering its legacy pain points, the strategic choice of a full‑stack reconstruction using Domain‑Driven Design, the new layered micro‑service architecture, data migration tactics, operational challenges, organizational safeguards, and the measurable performance gains achieved after launch.
Ctrip migrated its massive User Behavior Tracking system from ClickHouse to a compute‑storage separated StarRocks cluster on Kubernetes, achieving millisecond‑level query latency, halving storage usage, reducing node count, and sustaining millions‑of‑rows‑per‑second write throughput while simplifying scaling and operations.
Developers, data analysts, and ops engineers can eliminate the hassle of juggling multiple database clients by adopting DBeaver, an open‑source, cross‑platform tool that supports over 100 databases, offers powerful SQL editing, visual design, data migration, and backup features—all for free.
This article explains why data migration is the essential first step for cloud modernization, outlines the technical challenges of moving terabytes to petabytes, compares physical and logical migration methods, and presents practical solutions and real‑world case studies across Hive, cloud warehouses, lake‑house formats and analytic databases.
This guide details a step‑by‑step process for migrating a high‑traffic MongoDB app to a dedicated instance, covering full backup, real‑time incremental sync via Change Streams, compensating for missed changes, handling operation ordering, and ensuring zero downtime throughout.
Ctrip's User Behavior Tracking (UBT) system, handling 30 TB of daily data, moved from ClickHouse to StarRocks' compute‑storage separated architecture, cutting average query latency from 1.4 seconds to 203 ms, halving storage, reducing nodes from 50 to 40, and boosting write throughput to 3 million rows per second.
This article evaluates five common methods for moving data from cloud Elasticsearch to on‑premise INFINI Easysearch, details three viable solutions—Logstash, Elasticdump, and custom Python scripts—explains why reindex and snapshot restore fail, and provides a decision matrix and practical migration guidelines.
This guide walks through using the open‑source Elasticdump tool to migrate an Elasticsearch index—including mapping, settings, and data—to Easysearch, covering environment setup, SSL handling, step‑by‑step commands, verification methods, common errors, and performance‑tuning tips.
This guide walks through a zero‑downtime migration from Elasticsearch 9.0 to the Easysearch compatible branch using Logstash, covering project goals, solution selection, detailed input/filter/output configurations, SSL handling, batch tuning, execution steps, and verification to ensure data integrity and ID consistency.
This article details the end‑to‑end design, challenges, and implementation of a cross‑cloud migration of over 200 k Hive tables and nearly 40 PB of data using the self‑developed Kirk service, covering architecture, verification steps, and lessons learned to achieve 100 % data consistency without impacting production services.
Action OMS, a customized version of OceanBase’s OMS tool, enables seamless migration and real‑time data subscription from Microsoft SQL Server (2008‑2019 Enterprise) to OceanBase, detailing preparation, configuration, CDC setup, and best‑practice considerations for reliable, low‑latency data transfer.
This article details GoTerra's massive six‑month, 50 PB migration from GCP BigQuery to Alibaba Cloud MaxCompute, covering project scope, technical challenges such as complex data types, partition strategies, and high‑speed requirements, and explaining how the MaxCompute Migration Service (MMS) solved them with innovative architecture, scheduling, and data‑reorder techniques.
The article explains what a Redis big key is, why it harms performance, typical scenarios that generate big keys, and provides a step‑by‑step online optimization plan—including key sharding, dual‑write synchronization, progressive migration with HSCAN, gray‑scale traffic switch, non‑blocking deletion using UNLINK, and monitoring with rollback procedures—to answer the interview question confidently.
This article details the real‑world migration of Southeast Asian tech leader GoTerra from BigQuery to MaxCompute, showcasing how the open‑source dbt‑maxcompute adapter enables smooth ELT transitions, advanced incremental strategies, performance gains, ecosystem compatibility, and comprehensive best‑practice implementations for large‑scale data pipelines.
This article explains how to move data from an HBase cluster back into Kafka by extracting RowKeys with a MapReduce job, handling massive volumes and lack of partitioning, and reliably writing the records to Kafka while tracking successes and failures.
This guide outlines essential strategies for PHP developers to execute complex data migrations—covering thorough planning, incremental migration, transaction safety, dual‑write architectures, feature‑flag rollouts, real‑time syncing, and comprehensive monitoring—to ensure data integrity and keep system downtime to a minimum.
This article details the challenges of migrating 100,000+ SQL statements from BigQuery to MaxCompute for a leading Southeast Asian tech group, explains the new MaxCompute SQL syntax, auto‑partition and ingestion‑time tables, enhanced built‑in functions, and the BigQuery compatibility mode that ensures seamless query behavior.
This article walks through the end‑to‑end design, configuration, and implementation of splitting massive loan and repayment tables into 50 sharded MySQL tables using ShardingSphere, Spring Boot, and a custom suffix algorithm, while covering data migration, DBA coordination, dynamic switches, and scheduled consistency checks.
This article details the complex migration of a leading Southeast Asian tech group's data warehouse from Google BigQuery to Alibaba Cloud MaxCompute, outlining challenges such as storage format differences, SQL compatibility, and performance tuning, and explains how the new Append DeltaTable format with dynamic bucketing and incremental reclustering resolves these issues.
The article walks through why sharding is needed, outlines IO and CPU bottlenecks, presents two design principles, shows how to estimate capacity from existing data and growth trends, compares range, modulo, consistent‑hash and Snowflake sharding schemes, and details migration strategies for expanding nodes without downtime.
This article explores the performance bottlenecks of a 700‑million‑row MySQL orders table and presents a comprehensive sharding solution—including vertical and horizontal partitioning, gene‑based shard keys, routing logic, global secondary indexes, data migration strategies, and common pitfalls—demonstrating how query latency can drop from seconds to milliseconds while supporting billions of records.
Facing the challenge of moving 15 billion image files from Cassandra to S3, we iteratively redesigned the migration pipeline—from a single‑process approach to multi‑process, queue‑driven, and multi‑cluster deployments—reducing the projected 120‑day effort to just 40 days while ensuring reliability and performance.
This article walks through the complete process of designing, configuring, and coding a sharding solution for loan and repayment tables—including database basics, historical data migration, backend query refactoring, a three‑write strategy, scheduled consistency checks, and full SpringBoot/ShardingSphere implementation details.
This article explains how to migrate MySQL table data to Elasticsearch using a one‑time full sync, a scheduled incremental sync, addresses strong consistency challenges, and introduces Alibaba's Canal framework for reliable binlog‑based data replication.
This article explains the process of moving hash slots between nodes in a Redis cluster using the redis-cli --cluster reshard command, detailing the required parameters, interactive steps, and verification of successful slot migration.
A real‑world MySQL data‑migration case study shows how using INSERT INTO SELECT without proper indexing caused a full table scan, OOM errors, and data loss, and explains how to avoid the pitfall by adding appropriate indexes and understanding transaction locking.
This article explains the fundamental concepts of database sharding—including data nodes, logical and broadcast tables, sharding keys, strategies, algorithms, SQL parsing, routing, rewriting, execution, result merging, distributed primary keys, data masking, distributed transactions, data migration, and shadow databases—providing clear examples and code snippets for real‑world implementation.
This article presents a complete, step‑by‑step plan for migrating a legacy MongoDB‑based system to alternative data stores—including MySQL, Elasticsearch, and JD’s JImKV—while ensuring zero service interruption through careful scope analysis, DAO refactoring, dual‑write synchronization, gray‑release rollout, and robust monitoring and rollback mechanisms.
This article provides a comprehensive, step-by-step tutorial on using the open‑source Juicesync tool to migrate and synchronize object storage data across multiple cloud platforms, covering background, installation, configuration, full and incremental sync, and best‑practice notes.
This article shares a step‑by‑step guide for migrating billions of rows safely and quickly, covering divide‑and‑conquer batching, dual‑write architectures, tool selection, shadow testing, and rollback plans, with concrete Java and Spark code examples and practical pitfalls to avoid.
This article presents a complete MongoDB migration solution, detailing the migration rhythm, code refactoring using a decorator pattern, data source replacement with JImKV, MySQL and ES, bulk and incremental data transfer strategies, and deployment safeguards such as monitoring, gray release, and rollback to ensure a seamless cut‑over without service disruption.
This article explains how to migrate sharded Mycat data to ActionDB with ActionOMS, addressing duplicate primary keys by modifying both source and target schemas, configuring migration tools, and completing validation and post‑migration steps.
This article details the background, challenges, and technical solutions behind the partitioning redesign of the KPOS reporting system on OceanBase, covering deployment choices, parameter tuning, data migration, phased rollout, performance gains, encountered issues, and future scaling plans.
This article explains common data‑migration strategies such as hash‑modulo, time‑range and ID‑range sharding, and compares practical migration solutions including downtime migration, dual‑write, and an MQ + Redis approach, with detailed workflow diagrams and implementation tips.
OceanBase’s open‑source ecosystem—spanning data migration tools, developer utilities, a database proxy, and robust operational management—delivers comprehensive, open, and compatible support that enhances development, migration, and maintenance, positioning it as a competitive enterprise‑grade distributed relational database.
A developer’s costly mistake using MySQL’s INSERT INTO SELECT for data migration led to massive OOM errors, a full‑table scan, lost payment records, and ultimately a $10 000 loss and termination, highlighting the need for proper indexing and careful testing.
This article details the comprehensive process of testing business data migration in enterprise applications, covering requirements analysis, environment preparation, test planning, data validation, performance testing, security checks, rollback strategies, and feedback mechanisms to ensure accuracy and integrity.
This article presents Tongcheng Travel's metadata solution built on the open‑source Waggle Dance project, detailing the three‑layer architecture, challenges of a monolithic Hive Metastore, evaluated migration plans, federation implementation, migration workflow, and future directions for unified metadata governance.
This article describes the end‑to‑end design, configuration, and code implementation of a 50‑table sharding solution for loan and repayment request data, covering database schema planning, historical data migration, backend query adaptation, DBA coordination, SpringBoot 3 integration, common pitfalls, and a complete demo.
The author details both manual and one‑click migration of roughly 500 GB and 760 000 files from an older Intel‑based MacBook Air to a new M‑chip MacBook Pro, covering setup, SSH key handling, transferred settings, and a useful cursor‑speed command.
This technical article describes how a senior architect designed, configured, and migrated loan and repayment application tables to a sharded MySQL setup using ShardingSphere, Spring Boot, and MyBatis‑Plus, covering schema design, historical data sync, dual‑write switches, code integration, and common pitfalls.
This guide explains how to export MongoDB collections to JSON, load them into DuckDB, generate compatible table schemas, and then transfer the data efficiently into MySQL using DuckDB as an intermediate processing engine.
The article details how the vivo Game Center’s appointment service migrated billions of rows to a dedicated MySQL database with zero‑downtime by implementing a double‑write strategy via a custom MyBatis interceptor that mirrors updates and queries to the new database, handling full and incremental sync, primary‑key consistency, transaction limits, and final cut‑over steps.
The article introduces ActionOMS, a customized version of OceanBase Migration Service that adds automated performance and fault diagnosis, explains its architecture, showcases a real‑world case of Oracle‑to‑OceanBase synchronization, and demonstrates how the tool improves migration throughput and reduces latency.
The article details a zero‑downtime migration of a high‑traffic reservation service from a shared MySQL instance to an isolated database by employing a double‑write strategy implemented through a custom MyBatis plugin, covering full and incremental sync, consistency verification, traffic shifting, and key lessons on primary‑key handling and asynchronous write side‑effects.
This article details the design, planning, and step‑by‑step implementation of a database table‑partitioning solution for loan and repayment request tables, covering schema decisions, sharding strategy, historical data migration, dynamic synchronization, code examples with ShardingSphere, SpringBoot, and operational considerations for seamless migration.
After a year of business coding, the author details the design, implementation, and migration strategy for splitting loan and repayment tables into 50 sharded tables using ShardingSphere with Spring Boot, covering schema design, data synchronization, dynamic switches, and practical pitfalls.
This article details how Tencent Music’s data‑infrastructure team migrated thousands of ClickHouse and Druid nodes to a StarRocks compute‑storage‑separated lakehouse, achieving 40‑50% cost reduction while maintaining query performance, and shares the technical challenges, solutions, and best‑practice recommendations gathered during the process.
This technical report analyzes a high‑traffic MySQL deployment with three servers, detailing disk usage, table space, QPS and slow‑SQL issues, and presents a comprehensive governance plan that includes data migration, read‑traffic offloading to replicas, MyBatis interceptor implementation, JSF monitoring, and Python automation to reduce disk pressure and improve query performance.
This technical report describes the current high‑resource MySQL deployment, analyzes disk, table‑space, QPS and slow‑SQL issues, and presents a set of governance goals and concrete solutions—including data migration, query interception, and read‑from‑slave strategies—backed by code samples, tables and scripts to reduce load and improve stability before a major sales event.
This guide walks through installing JDK and MySQL on Windows, configuring environment variables via batch scripts, adjusting Nginx paths, and using offline installation methods so that the entire stack—including data—can be copied to other machines for rapid, repeatable deployment.
The article describes how a one‑year‑old loan/repayment service was refactored by introducing a 50‑table sharding scheme with ShardingSphere, detailing the design decisions, historical data migration, code changes, configuration files, pitfalls, and runtime synchronization strategies to achieve efficient query performance on billions of rows.
The article outlines a cross‑cluster upgrade to StarRocks 2.5.13, evaluating resource and stability costs, and presents two migration schemes—using external tables and a Flink connector—along with planning, parallel execution, validation steps, and results showing successful migration of over 10 TB at 2 Gb/s across ten nodes, while noting future automation and CDC enhancements.
This article presents a comprehensive guide to designing and implementing a Java‑based MySQL large‑table migration tool using microservice architecture, multiple design patterns, multithreading, custom logging, and flexible migration modes, complete with code snippets, repository links, and deployment considerations.
This article details how upgrading from Elasticsearch 5.x/2.x to 8.x dramatically improves search, aggregation, and write performance while reducing storage costs, introduces vector KNN, synthetic _source, TSDS, searchable snapshots, security enhancements, and provides migration examples and code snippets for enterprise search platforms.
This article recounts a costly MySQL data‑migration failure caused by an unchecked INSERT INTO SELECT operation, analyzes why full‑table scans and transaction isolation led to payment‑record loss, and offers practical safeguards such as proper indexing and careful use of the statement.
This article recounts a real-world MySQL data migration incident where using INSERT INTO SELECT caused out‑of‑memory errors and data loss, analyzes why full table scans and transaction locking led to failures, and offers practical recommendations such as indexing and avoiding bulk inserts for large tables.
Database PerformanceINSERT INTO SELECTdata migration
This article explains common data‑quality challenges when rebuilding business systems, compares manual SQL‑based validation with a dedicated data‑comparison product, and walks through practical steps for configuring, executing, and reviewing automated data‑matching tasks in a big‑data environment.
A real‑world MySQL post‑mortem shows how using INSERT INTO SELECT for large‑scale data migration caused full‑table scans, transaction‑level locking, data loss and a costly dismissal, and explains how proper indexing and query design can prevent such disasters.
A MySQL‑driven data‑migration using INSERT INTO SELECT caused full‑table scans, lock contention and lost payment records, leading to a $10,000 loss and the engineer’s dismissal; the post‑mortem explains the root cause and how proper indexing can prevent it.
A large‑scale MySQL table migration using INSERT INTO SELECT caused hidden full‑table scans and lock contention, leading to data loss and a costly OOM incident, but adding the proper index and understanding transaction isolation prevented the failure.
The article analyzes a real‑world MySQL data‑migration failure caused by an INSERT INTO SELECT operation that triggered full‑table scans and row‑level lock contention, leading to massive data loss and highlighting the need for proper indexing and cautious use of this command.
A MySQL data‑migration task using INSERT INTO SELECT triggered a full‑table scan, leading to lock contention, payment‑record loss, and ultimately the developer's dismissal, highlighting the dangers of unindexed bulk operations and the need for proper indexing and testing.
A real‑world MySQL case study shows how using INSERT INTO SELECT for large‑scale data migration without proper indexing caused full‑table scans, lock contention, and data loss, highlighting the need for careful query planning and performance testing.
This article presents a comprehensive analysis of the ZhaiZhai financial system's growing data volume and slow‑query issues, evaluates four storage solutions—including sharding, hot‑cold separation, TiDB, and OceanBase—selects TiDB as the optimal choice, and details the migration steps and Elasticsearch integration to improve performance and scalability.
This article explores strategies for seamless data migration—including full, incremental, and binlog‑based approaches—and examines distributed transaction models such as XA, BASE, TCC, and AT, outlining their components, workflows, advantages, challenges, and supporting tools like Seata and Canal.
WCDB 2 introduces multi-language support (C++, Java, Kotlin), a rewritten SQL builder, fast backup and recovery with sub‑second times, incremental WAL‑based backup, automatic data migration, Zstandard compression, on‑the‑fly column addition, and performance enhancements such as an FTS5 optimizer and interruptible transactions, delivering a robust high‑performance mobile database solution.
This article shares Xiaomi's real‑world challenges and solutions when building a new Spark 3.1‑based data platform, covering Multiple Catalog implementation, Hive‑to‑Spark SQL migration, automated batch upgrades, performance and stability optimizations, and future roadmap for vectorized execution.
This guide introduces Strapi, an open‑source headless CMS, covering its key features, quick installation, language switching, basic usage for creating content types, exposing RESTful and GraphQL CRUD APIs, visualizing data with external tools, and migrating SQLite databases to other systems.
This article outlines a six‑stage, progressive migration plan for replacing legacy databases with AntDB, covering deployment, extensive testing, SQL refactoring, data migration, go‑live procedures, and monitoring, while providing detailed tuning recommendations for dozens of critical database parameters.
This guide explains MariaDB ColumnStore’s columnar architecture, key advantages, step‑by‑step installation, field and SQL limitations, how to migrate InnoDB data, configure multi‑source replication, compare performance with InnoDB, and achieve high availability and dynamic scaling for analytical workloads.
This article introduces big data testing concepts, outlines common testing methods, evaluates the Python library Datacompy against alternatives, and details a practical implementation for large-scale data migration and validation, including configuration, volume comparison, content verification, and performance optimizations such as sorting and multithreading.
This article explains how to migrate a specific Redis database (db8) to another server's db15 by writing a Python script that batches keys, tracks progress, estimates remaining time, and verifies the migration results.
This article presents a detailed technical case study on expanding and optimizing a system's storage capacity, covering business background, current architecture, implementation plans, technology selection, data synchronization strategies, phased rollout steps, results, and remaining challenges, with concrete metrics and diagrams.
A real‑world MySQL case study shows how using INSERT INTO SELECT for large data migrations can trigger full‑table scans, lock contention, and massive data loss, emphasizing the need for proper indexing and careful testing before deployment.