How We Migrated MySQL to Tencent Cloud CDB and Boosted Performance Up to 10×
This case study details the migration of Weimeng's MySQL databases to Tencent Cloud CDB, describing the testing methodology, performance bottlenecks discovered (NUMA, network parameters, low‑concurrency issues, and version bugs), the step‑by‑step optimizations applied, and the resulting QPS improvements across various workloads.
Background : After a severe outage and data‑recovery incident, Weimeng's DBA team decided to replace their self‑managed MySQL instances with Tencent Cloud CDB (TencentDB for MySQL) to improve data safety, high availability, and performance.
Test Setup
Configuration: Shanghai‑4 zone, 16C 64G CDB instances. Benchmark tables were built from production data, and 27 production SQL workloads were collected for business‑level testing.
Control groups: Blackstone (unlimited CPU, 56‑core) and Blackstone limited to 16‑core.
CDB variants: native CDB, CPU‑optimized, network‑parameter‑optimized, MySQL‑parameter‑optimized.
Baseline stress tests at concurrency levels 16, 32, 64, 100, 200, 400, measuring QPS.
Business stress tests using the same concurrency levels.
Conclusion : Optimized CDB meets the company’s database demand.
Verification Method
Stress testing was performed with mysqlslap:
mysqlslap --iterations=100 --create-schema='test' --query="query.sql" --number-of-queries=20000 --delimiter=";" --concurrency=100QPS was calculated as 20000 / average_time. Multiple optimization rounds were carried out from April 8 to April 26, addressing three major performance regressions.
Key Optimization Areas
1. NUMA & CPU Isolation
Problem: CDB used cgroup CPU isolation, causing cross‑NUMA memory accesses and higher lock contention.
Solution: Align CDB CPU isolation with self‑built servers (bind cores to the same NUMA node) and adjust cpu_quota.
Result: 10× performance gain, CDB surpasses self‑managed MySQL.
2. Network Parameters
Problem: High concurrency caused performance drop due to many short‑lived connections.
Solution: Increase TCP buffers and backlog settings:
tcp_rmem="4096 873800 4194304"
tcp_wmem="4096 163840 4194304"
tcp_max_syn_backlog=3240000
somaxconn=2048Result: Resolved high‑concurrency slowdown.
3. Low‑Concurrency Network Latency
Problem: CDB traffic traversed an extra TGW (gateway) layer, adding latency for low‑concurrency queries.
Solution: Plan to remove TGW, shortening the path to CVM‑>VPCGW‑>CDB.
Result: Low‑concurrency performance gap eliminated.
4. Range‑Query Regression in CDB 5.6
Problem: A commit introducing read‑transaction‑id 0 broke innodb_rows_read counting, degrading performance.
Solution: Patch the kernel and release a new CDB 5.6 version.
Result: New version outperforms self‑managed MySQL.
5. Parameter Tuning for 100‑Concurrency QPS Gap
Problem: Even after earlier fixes, CDB QPS at 100 concurrency lagged behind.
Solution: Adjust MySQL variables (e.g., innodb_io_capacity_max, innodb_write_io_threads, table_open_cache, etc.) using a tool like CDBTune.
Result: Overall performance improved markedly, many instances now exceed self‑managed QPS.
Migration Process
Configure CDB instances based on QPS requirements (e.g., 16C64G for core services).
Plan migration per instance with owners, schedule, and DTS tasks.
Execute migration: create DTS jobs, verify data consistency, switch DNS, stop source DB, and validate connections.
Final Outcomes
Baseline QPS increased 4‑10× after optimizations.
Business‑level queries improved >2× in many scenarios.
Performance variance stayed within 25 % across most workloads.
Post‑Migration Recommendations
Assess QPS and choose appropriate CDB specs.
Run comparative stress tests before migration.
Match CDB version and parameters with the source DB.
Document migration steps and share updates across teams.
Verify account permissions and perform a final data‑consistency check.
Automate migration tasks to reduce downtime.
Run ANALYZE TABLE after DTS to refresh statistics.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
ITPUB
Official ITPUB account sharing technical insights, community news, and exciting events.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
