Tagged articles
large datasets
10 articles · Page 1 of 1
FastAPI feels lightning‑fast on small datasets, but returning millions of rows can exhaust memory, block the event loop, and cripple the database; this article explains why that happens and provides concrete design rules—selective fields, pagination, cursor‑based queries, streaming, and chunked processing—to keep APIs stable at scale.
FastAPIPaginationlarge datasets
0 likes · 9 min read

This tutorial demonstrates how MySQL LIMIT pagination performs on tables with millions of rows, measures query times for various offsets and batch sizes, and presents practical optimization techniques such as sub‑queries, ID‑range filtering, and column selection to improve speed.
LIMITMySQLPagination
0 likes · 9 min read

This article explains why data cleaning and exploratory analysis consume most of a data scientist's time, introduces the pandas‑profiling library as a richer alternative to pandas.describe(), shows how to install and generate reports, customize them via code or YAML, and discusses performance considerations for larger datasets.
EDAdata analysislarge datasets
0 likes · 8 min read

This article walks through generating tens of millions of rows in MySQL, then demonstrates a series of optimizations—including simple and covering indexes, data reduction, small‑table‑driven joins, and forced indexes—showing how each technique impacts query performance and EXPLAIN output.
EXPLAINIndexingMySQL
0 likes · 14 min read

This article walks through practical techniques for removing duplicates from massive Python datasets—such as using tuples with sets, merging iteratively, and concatenating with drop_duplicates—while highlighting memory pitfalls and offering concise code snippets.
big datadata deduplicationlarge datasets
0 likes · 3 min read

Through hands‑on experiments with a 10‑million‑row MySQL 5.7 table, this article shows how standard LIMIT pagination becomes slower as offset and data volume grow, then presents practical optimizations—sub‑queries, ID‑based ranges, and column selection—to dramatically improve query performance.
MySQLlarge datasets
0 likes · 11 min read

This article shows how to generate tens of millions of rows in MySQL, measure insertion and pagination performance, and apply practical optimizations such as sub‑queries for large offsets and column selection to dramatically speed up queries on huge data sets.
MySQLPaginationlarge datasets
0 likes · 10 min read

This article demonstrates how to generate, insert, and query ten‑million‑row MySQL tables, measures pagination query times under different offsets and limits, and presents practical optimization techniques such as sub‑queries, id‑range filtering, and column selection to improve performance on massive data sets.
MySQLPaginationSQL
0 likes · 10 min read

This article investigates why MySQL queries with large LIMIT offsets become extremely slow, demonstrates the issue with simulated millions‑of‑row datasets, and presents three optimization strategies—including index‑covering subqueries, remembering the last primary‑key position, and applying offset throttling—to achieve consistent, fast pagination performance.
IndexingMySQLPagination
0 likes · 12 min read


ITPUBSep 7, 2016 · Databases
This guide explains how to clean redundant fields, restructure a massive MySQL table, batch‑fetch data with LIMIT, rewrite slow queries, and compare three insertion methods—including prepared statements and bulk inserts—to cut migration time from hours to under 20 minutes.
MySQLSQL Optimizationbatch processing
0 likes · 7 min read
