This article analyzes the constraints of loading a billion 1‑KB log records from HDFS/S3 into MySQL, evaluates single‑table limits, proposes batch inserts, sharding, file‑reading techniques, task coordination with Redis, Redisson, and Zookeeper, and offers practical performance‑tuning recommendations.
This article demonstrates how to insert 300,000 rows into a MySQL table using MyBatis and JDBC, compares direct batch insertion, row‑by‑row insertion, and optimized batch strategies, and provides performance measurements and practical tips for efficient large‑scale data loading.
This tutorial explains how to use PHP and PDO to quickly insert ten million rows into a remote MySQL database by preparing the connection, employing batch inserts, managing transactions, tuning MySQL settings, and optionally using the LOAD DATA statement for optimal performance.
This article documents practical experiments comparing four MySQL bulk‑insert techniques—single‑row loop, concatenated SQL, MyBatis‑Plus saveBatch, and loop with batch mode—in a SpringBoot environment, revealing the most efficient approach and key configuration tips.
This article evaluates single‑threaded bulk‑insert performance in MySQL by testing various batch sizes, analyzes memory usage and transaction merging, and provides practical recommendations for efficiently generating millions of records.
This guide shows how to configure a Spring Boot 2.3.12 application with Elasticsearch 7.8, create index templates, implement a reusable bulk operator service, and test bulk insertion of task data into dynamically named indices, ensuring automatic index creation and proper mapping.
This article presents practical MySQL optimization techniques—including disabling keys for MyISAM, sorting InnoDB input, adjusting UNIQUE_CHECKS and AUTOCOMMIT, using multi‑row INSERT, LOAD DATA INFILE, ORDER BY NULL for GROUP BY, index‑aware ORDER BY, OR condition handling, and SQL hints—to dramatically speed up bulk data loads and common queries.
This article presents practical MySQL performance tricks, covering bulk data insertion with DISABLE/ENABLE KEYS, loading data in primary‑key order, multi‑value INSERTs, INSERT DELAYED, separating index and data files, GROUP BY sorting elimination, index‑driven ORDER BY, OR‑condition indexing, replacing sub‑queries with JOINs, and using SQL hints such as USE, IGNORE, and FORCE INDEX.
This article explains how to achieve high‑performance bulk inserts in .NET Core applications by converting entities to CSV or DataTable formats and using MySQL's LOAD DATA and SQL Server's SqlBulkCopy, providing reusable extension methods, interface abstractions, and performance benchmarks.
This article presents a comprehensive guide to MySQL performance tuning, covering bulk‑insert optimizations, efficient INSERT statements, GROUP BY and ORDER BY improvements, OR‑condition handling, nested query replacement, and the use of SQL hints such as USE INDEX, IGNORE INDEX, and FORCE INDEX, with practical code examples and benchmark results.
This article walks through a real‑world monitoring project that must store over 400 million rows per day in SQL Server 2012, detailing the performance bottlenecks, bulk‑copy tuning, schema redesign, partitioning strategies, index removal and creation, and query‑optimisation techniques that finally achieve sub‑second query times and meet strict latency requirements.
This article explains why frequent small commits are slower than a single bulk SQL operation, demonstrates how to rewrite PL/SQL procedures to use bulk inserts, adjust UNDO settings, and enable parallel execution, achieving up to six‑fold performance gains on large data loads.
This guide explains why MySQL INSERT operations can be slow, breaks down the time‑consuming factors, and provides a collection of practical techniques—such as multi‑row VALUES, LOAD DATA INFILE, table locking, and special modifiers like DELAYED, IGNORE, and ON DUPLICATE KEY UPDATE—to dramatically speed up bulk data insertion.
To meet a demanding monitoring project that required storing up to 400 million records per day in SQL Server 2012, the author details a step‑by‑step journey involving bulk‑copy insertion, removing indexes during load, hourly partitioned tables, strategic index creation, and query optimizations to achieve sub‑second query times.
