This article analyzes the constraints of loading a billion 1‑KB log records from HDFS/S3 into MySQL, evaluates single‑table limits, proposes batch inserts, sharding, file‑reading techniques, task coordination with Redis, Redisson, and Zookeeper, and offers practical performance‑tuning recommendations.
This article explains how to import 1 billion 1 KB log records stored in HDFS/S3 into MySQL by analyzing single‑table limits, proposing sharding into multiple tables, choosing the right storage engine, using batch inserts, controlling concurrency, and ensuring ordered, reliable processing with Redis and Redisson.
This article explains how to handle massive Excel import/export tasks in Java by comparing POI workbook implementations, selecting the appropriate class based on data size, and using EasyExcel with batch processing, JDBC transactions, and pagination to achieve fast, low‑memory operations for hundreds of millions of rows.
This article explains how to import a hundred‑million‑row Excel dataset into MySQL by using CSV format, streaming parsers like EasyExcel, batch inserts, asynchronous processing, and partial‑transaction strategies to ensure feasibility, data integrity, and high performance.
The article explains how SSH idle timeout, network drops, or closing the terminal send a SIGHUP signal that aborts MySQL import scripts, and it provides step‑by‑step solutions using nohup, tmux/screen, and systemd along with practical tips for reliable large‑scale data loading.
This tutorial explains how to use the Python PrettyTable library to generate well‑formatted command‑line tables, covering installation, row and column insertion, data import from CSV, databases and HTML, various output formats, selective printing, sorting, and custom styling options.
This guide outlines the StarRocks SRCA certification exam, highlights essential study resources, breaks down the core topics such as architecture, data import/export, SQL optimization and performance tuning, and offers practical tips, mock‑exam details, and personal experience to help candidates succeed.
This article explains how to import 1 billion 1 KB log records stored in HDFS or S3 into MySQL by analyzing single‑table limits, using batch inserts, choosing storage engines, sharding, optimizing file‑reading methods, and coordinating distributed tasks with Redis, Redisson, and Zookeeper to ensure ordered, reliable, and high‑throughput data loading.
This article explains how to handle massive Excel‑DB import/export tasks in Java by comparing POI workbook types, selecting the right implementation, and leveraging EasyExcel with batch queries, sheet splitting, and JDBC batch inserts to process over three million rows efficiently.
This guide explains how to use Pandas to import data from various file formats such as CSV, Excel, JSON, SQL, HTML, HDF5, Pickle, TSV, fixed‑width files and the clipboard, and also demonstrates how to export DataFrames to formats like CSV, Excel, JSON, SQL, HTML, HDF5 and Pickle, providing clear code examples for each operation.
TDE‑ClickHouse, the core engine of Baidu MEG’s Turing 3.0 ecosystem, delivers sub‑second, self‑service analytics on petabyte‑scale data by decoupling compute, adding multi‑level aggregation, high‑cardinality and rule‑based optimizations, a two‑phase bulk‑load pipeline, cloud‑native deployment, and a lightweight meta service, now powering over 350 000 cores, 10 PB storage and more than 150 000 daily BI queries with average response times under three seconds.
This article presents ten carefully crafted PHP project concepts—from building a custom MVC framework and implementing data import/export to creating QR code generators and event management tools—each designed to showcase backend expertise and enhance a developer's professional portfolio.
This article analyzes how to import one billion 1 KB log records stored in HDFS or S3 into MySQL by evaluating single‑table limits, choosing storage engines, designing sharding, batch insertion, file‑reading strategies, task coordination, and distributed locking to achieve high‑throughput and ordered writes.
This guide explains how to efficiently load one billion 1‑KB log entries from HDFS or S3 into MySQL by analyzing B‑tree limits, using batch inserts, choosing the right storage engine, sharding tables, optimizing file reading, and coordinating tasks with Redis, Redisson, and Zookeeper.
This guide walks you through using StarRocks—covering effortless expression‑based partitioning, streamlined data loading with INSERT FROM FILES and PIPE, powerful in‑flight data transformation using SELECT/JOIN/UNNEST, and flexible table structure tweaks via ALTER TABLE to boost query performance.
To import one billion 1 KB log records stored in HDFS or S3 into MySQL as quickly as possible, the article analyzes constraints, evaluates single‑table limits, proposes sharding, batch inserts, storage‑engine choices, file‑reading strategies, task coordination, reliability mechanisms, and concurrency control.
This article walks through the full engineering process of importing a billion 1 KB log records into MySQL, covering data size calculations, B‑tree index limits, batch‑insert strategies, storage‑engine trade‑offs, fast file‑reading techniques, task coordination with Redis and Redisson, and distributed‑lock design using Zookeeper.
To import one billion 1 KB log records into MySQL efficiently, the article examines data size constraints, B‑tree index limits, batch insertion strategies, storage engine choices, file‑reading techniques, task coordination with Redis, Redisson semaphores, and distributed lock handling to ensure ordered, reliable, high‑throughput loading.
This article analyzes the constraints of loading one billion 1 KB log records from distributed storage into MySQL, evaluates single‑table limits, proposes batch‑insert, sharding, storage‑engine, file‑reading, and distributed‑task coordination techniques to achieve high‑speed, ordered, and reliable data ingestion.
To import one billion 1 KB unstructured log entries stored in HDFS or S3 into MySQL efficiently, the article discusses constraints, B+‑tree limits, batch insertion, storage engine choices, sharding, task coordination, concurrency control, progress tracking with Redis, and reliable distributed execution strategies.
This article compares ByConity and ClickHouse from a usage perspective, detailing their architectural differences, core components, basic operations such as table creation, data import and query, distributed transaction support, special table engines, scaling strategies, and deployment requirements.
This article explains how to import 1 billion 1 KB log records stored in HDFS or S3 into MySQL by analyzing table capacity limits, choosing storage engines, designing batch inserts, coordinating file reading and writing, and handling task reliability with Redis, Redisson, and Zookeeper.
This article compares various MySQL data import/export methods—including mysqldump, mydumper, SELECT OUTFILE, and MySQL Shell utilities—explaining their commands, configuration options, performance results, and best‑practice recommendations based on testing with a 10‑million‑row table.
This article explains Doris's various data import methods, focuses on the routine load approach for Kafka streams, describes how to handle mixed‑schema topics using the max_error_number parameter, and compares query performance of a 130 million‑row dataset against ClickHouse, highlighting each system's strengths and limitations.
This article demonstrates how to ingest billions of Kafka records into Doris using Routine Load, compares the import process and query performance with ClickHouse’s Kafka engine tables, discusses configuration challenges such as field‑count filtering, and presents practical workarounds and benchmark results.
This article explains the differences between POI's HSSFWorkbook, XSSFWorkbook and SXSSFWorkbook, provides guidelines for selecting the right workbook based on data volume and Excel features, and demonstrates batch export and import of hundreds of millions of rows using EasyExcel, JDBC batch inserts, and transaction control to achieve high performance and low memory consumption.
This guide introduces DataGrip, a JetBrains database client, covering download, installation, theme selection, driver management, connection setup, SQL editing shortcuts, data export/import, table creation, navigation, and various productivity features for efficient database development.
This article explains how to efficiently import and export massive Excel datasets in Java by comparing POI implementations, selecting the appropriate workbook type, and using EasyExcel with batch processing, pagination, and JDBC transactions to achieve high performance for hundreds of thousands to millions of rows.
This guide compiles a comprehensive set of MySQL command‑line scripts for exporting whole databases, individual tables or schemas, importing data, and performing common database and table operations such as creation, selection, alteration, deletion, granting privileges, and viewing structure, all illustrated with ready‑to‑use code examples.
This guide explains the underlying snapshot and transaction‑log storage of ZooKeeper, outlines three key scenarios for using MSE’s import‑export feature, and provides step‑by‑step instructions with code snippets to back up, migrate, or analyze ZooKeeper clusters efficiently.
MySQL 8.0.31 introduces a new syntax that allows importing pre‑computed histogram data directly into a table, dramatically reducing the time needed to update histogram statistics compared with the traditional online creation method.
This article explains the challenges of large‑scale Excel import/export in Java, compares POI workbook implementations, introduces EasyExcel for high‑performance handling of hundreds of thousands to millions of rows, and provides detailed code examples for batch querying, sheet management, and JDBC batch insertion with transaction control.
This article explains how to handle massive Excel import and export tasks in Java by comparing traditional POI implementations, selecting the appropriate Workbook type based on data volume, and leveraging Alibaba's EasyExcel library together with batch JDBC operations to process over three million rows with minimal memory usage and high performance.
This tutorial introduces the Pandas library, covering installation, data import from CSV and Excel, DataFrame creation, descriptive statistics, indexing with loc/iloc, and applying custom functions to clean and transform column values, all illustrated with code snippets and images.
This article analyzes traditional POI implementations, compares their strengths and weaknesses, and presents a high‑performance solution using EasyExcel for exporting and importing up to 3 million rows, including batch querying, sheet management, JDBC batch inserts, and detailed performance benchmarks.
This article explains how to handle massive Excel import and export tasks in Java by comparing POI's HSSFWorkbook, XSSFWorkbook, and SXSSFWorkbook, selecting the right implementation, and using Alibaba's EasyExcel together with batch queries and JDBC batch inserts to process three million rows efficiently without memory overflow.
This article explains how to handle large‑scale Excel import and export in Java by comparing traditional POI workbooks, selecting the appropriate implementation based on data volume and requirements, and presenting a high‑performance solution with EasyExcel, batch processing, and JDBC transactions for hundreds of millions of rows.
This article provides a detailed tutorial on JetBrains DataGrip, covering installation, driver management, connection configuration, SQL editing shortcuts, data import/export, schema visualization, and various productivity features for efficiently working with relational databases.
This tutorial walks through installing Pandas, importing CSV and Excel files, building DataFrames from dictionaries, describing data, indexing with loc/iloc, and applying custom functions to transform columns, providing clear code examples and visual outputs.
This article explains the DBLE split feature that partitions large mysqldump files according to sharding configuration, provides command syntax and usage examples, and presents a performance test showing that split‑based import can be up to 18 times faster than direct DBLE import while preserving data correctness.
This guide provides step‑by‑step instructions for downloading Apache Doris, configuring and deploying FE, BE, and Broker nodes, performing scaling operations, managing users and tables, importing and exporting data, and integrating Doris with Spark and Flink using code examples.
This tutorial walks you through installing Pandas, importing data from CSV and Excel, creating DataFrames from dictionaries, describing datasets, indexing with loc/iloc, and cleaning columns using apply, all illustrated with clear code examples and visual outputs.
This article introduces Alibaba's EasyExcel library, explains its memory‑efficient design, lists its main advantages and common annotations, provides Maven dependencies, and demonstrates import and export code snippets for Java backend applications.
This tutorial explains how to set up Composer, install the PHPExcel library, create a front‑end upload form with Layui, write JavaScript for file handling, and implement a ThinkPHP 5.1 backend that reads Excel files and imports member data into the database.
This article explains why using MySQL's --force option with default extended‑insert backups can cause missing rows during import, demonstrates the issue with a reproducible example, and shows how disabling extended‑insert with --skip-extended-insert restores all data while providing practical usage recommendations.
This article explains how to design and implement a fully elastic, zero‑ops Serverless database using Alibaba Cloud Tablestore, covering its architecture, key features, step‑by‑step CLI setup, table creation, data import, querying, indexing, and cost efficiency for high‑volume order workloads.
This guide explains why CSV files appear garbled in Excel, then walks you through two practical solutions—opening the file with Notepad++ using UTF‑8 encoding and importing it via Excel’s “From Text/CSV” wizard—complete with screenshots and code examples.
This article presents a detailed technical comparison between Apache Doris and ClickHouse, covering their architecture, deployment, distributed capabilities, transaction support, data import, storage design, query performance, cost, and future development, and provides guidance on selecting the appropriate engine for specific OLAP scenarios.
This article compares MySQL's INSERT INTO and LOAD DATA INFILE statements, explains their syntax, shows practical examples, highlights when each should be used for bulk data loading, discusses error handling with IGNORE, and explains why LOAD DATA INFILE is typically faster.
This article presents a complete MySQL interview guide, including a script to create and populate tables for students, teachers, courses, scores, and grades, followed by 33 common interview questions with detailed SQL solutions covering selection, aggregation, joins, subqueries, and ordering.
This article provides a step‑by‑step walkthrough of installing DataGrip, configuring data sources, using its SQL console, creating tables, exporting and importing data, and leveraging powerful navigation and editor shortcuts to boost database productivity.
This article introduces MySQL Shell’s import_table utility, demonstrates its core features and configuration options such as parallel import, speed throttling, and custom chunk sizes, provides side‑by‑side command examples with LOAD DATA, and shows that import_table can achieve more than double the import speed in practical tests.
This article explains how to import data between Laravel projects using chunkById, and demonstrates a technique to directly access a model's protected "attributes" property by casting the object to an array and using the chr(0) prefix, eliminating manual assembly of the data array.
The article explains how MySQL 8.0.22 enhances the util.import_table function with the decodeColumns option, allowing users to preprocess each row during import, replace load data statements, and customize column values directly within MySQL Shell.
This article provides a comprehensive guide to using Sqoop for importing data from relational databases into HDFS, Hive, and HBase, as well as exporting data back to databases, covering command syntax, options, and practical examples for big‑data workflows.
This tutorial demonstrates how to import CSV and Excel stock‑listing files using pandas, adjust parsing parameters, view data summaries, and combine multiple sheets into a single DataFrame through concatenation, providing a practical example of data import and merging in Python.
This tutorial walks through creating a TI-ONE Notebook instance, importing competition data, training a TensorFlow model, and uploading results, providing a complete end‑to‑end workflow for participants of the 2020 Tencent Advertising Algorithm Competition.
This step‑by‑step guide shows how to download the FIFA 18 player dataset from GitHub, create a MySQL database and table, read the CSV with Python’s csv.DictReader, generate INSERT statements, batch‑load the data efficiently, and query Argentine players, illustrating a complete end‑to‑end data‑import workflow.
This article explains how DBLE implements the MySQL LOAD DATA large‑scale data import feature, covering the overall design, key classes such as ServerLoadDataInfileHandler and LoadDataUtil, and detailed Java source code handling client‑server interactions, file routing, and backend MySQL communication.
This tutorial demonstrates how to use Python to read dozens of txt files, write millions of records into a MySQL database, clean duplicate entries, and export selected rows to new files, offering a fast, scriptable alternative to manual Excel processing.
Youzan’s evolution of HBase bulk‑load—from manual MapReduce jobs to Hive‑SQL and finally Spark—demonstrates how generating HFiles on HDFS, partitioning by region, sorting keys, and handling serialization issues enables billions of records to be loaded efficiently without disrupting production clusters.
This article explains how Java backend developers can replace memory‑heavy Apache POI with EasyExcel for fast, low‑memory Excel imports, covering basic reading, custom converters for gender and dates, and multiple Spring Boot integration patterns including controllers, listeners, constructor injection and lambda‑based approaches.
This article demonstrates how to use MySQL Shell's util.import_table and util.import_json functions to import CSV and JSON data efficiently, compares their performance with the traditional mysqlimport tool, explains required options such as file path, schema, table, dialect, threading, and shows example sessions in both JavaScript and Python modes.
This article shares a real‑world case study of loading billions of rows from an unsplittable text file into an Oracle table using SQL*Loader, detailing a parallel skip‑and‑load technique, performance results, a critical 4‑byte row limit bug, and practical recommendations for CPU‑vs IO‑bound workloads.
This article details the architecture and operational enhancements of 58.com’s HBase platform, covering multi‑tenant resource isolation, various data access APIs, bulk import/export mechanisms, and a series of performance optimizations that improve stability and scalability for massive data workloads.
This tutorial demonstrates how to create a MySQL database and table, import data from /etc/passwd, add an auto‑increment primary key, export selected records to a file, and perform common SQL queries, conditions, and table‑record manipulations such as insert, update, and delete.
This article explains practical methods for importing various external data formats—such as CSV, plain text, XML, and JSON—into MySQL, covering command-line utilities, LOAD DATA statements, stored procedures, third‑party tools like mysqljsonimport, and using MySQL Workbench, while addressing configuration and security considerations.
