Apr 8, 2024 · Databases

Designing High‑Performance Import of 1 Billion Records into MySQL

This article analyzes how to import one billion 1 KB log records stored in HDFS or S3 into MySQL efficiently, covering constraints, single‑table limits, batch insertion, storage‑engine choices, sharding, file‑reading techniques, task reliability, and concurrency control to achieve optimal throughput.

Batch InsertDistributed TasksLarge Data Import

0 likes · 19 min read

Designing High‑Performance Import of 1 Billion Records into MySQL

ITPUB

Jul 5, 2019 · Databases

How to Load 2 Billion Rows into MySQL Fast with TokuDB: Performance Test

A friend needed to import over 2 billion records from a big‑data platform into MySQL for next‑day reporting, so we evaluated InnoDB versus XeLabs TokuDB, configured TokuDB parameters, created test tables, used LOAD DATA, measured write speed, file size, and discovered TokuDB can achieve about 570 k rows per second, roughly three to four times faster than InnoDB, especially when avoiding auto‑increment columns.

Large Data ImportTokuDBbulk load

0 likes · 8 min read

How to Load 2 Billion Rows into MySQL Fast with TokuDB: Performance Test