Skip to content
Tagged articles

Large Data Import

2 articles · Page 1 of 1
Selected Java Interview Questions
Selected Java Interview Questions
Apr 8, 2024 · Databases

Designing High‑Performance Import of 1 Billion Records into MySQL

This article analyzes how to import one billion 1 KB log records stored in HDFS or S3 into MySQL efficiently, covering constraints, single‑table limits, batch insertion, storage‑engine choices, sharding, file‑reading techniques, task reliability, and concurrency control to achieve optimal throughput.

Distributed TasksLarge Data ImportPerformance Tuning
0 likes · 19 min read
Designing High‑Performance Import of 1 Billion Records into MySQL
ITPUB
ITPUB
Jul 5, 2019 · Databases

How to Load 2 Billion Rows into MySQL Fast with TokuDB: Performance Test

A friend needed to import over 2 billion records from a big‑data platform into MySQL for next‑day reporting, so we evaluated InnoDB versus XeLabs TokuDB, configured TokuDB parameters, created test tables, used LOAD DATA, measured write speed, file size, and discovered TokuDB can achieve about 570 k rows per second, roughly three to four times faster than InnoDB, especially when avoiding auto‑increment columns.

Large Data ImportMySQLTokuDB
0 likes · 8 min read
How to Load 2 Billion Rows into MySQL Fast with TokuDB: Performance Test