BestHub
Sign in
Tagged articles

Batch Processing

251 articles · Page 2 of 3
JD Retail Technology
JD Retail Technology
Feb 29, 2024 · Databases

Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution

This article presents a real‑world case study of optimizing massive batch‑processing tasks in an ad‑platform by applying query‑level improvements, cursor‑based pagination, shard‑aware batch updates, JVM‑tuned garbage collection, and distributed load‑balancing, ultimately reducing CPU usage from 80% to under 2% and cutting query‑per‑minute volume from millions to a few thousand.

Batch ProcessingCursor PaginationSharding
0 likes · 22 min read
Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution
Python Crawling & Data Mining
Python Crawling & Data Mining
Feb 27, 2024 · Fundamentals

Automate Multi‑Sheet Excel Scoring with Python & Pandas: Step‑by‑Step Guide

This article walks through using Python and pandas to batch‑process seven Excel evaluation sheets, skipping header rows, cleaning data, computing total and average scores per person, merging results, and outputting aggregated statistics, providing a practical automation solution for repetitive office tasks.

Batch ProcessingData ProcessingExcel automation
0 likes · 7 min read
Automate Multi‑Sheet Excel Scoring with Python & Pandas: Step‑by‑Step Guide
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Feb 24, 2024 · Backend Development

Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing

This article introduces Karta, a lightweight Go library that provides two modes—Pipeline for unknown‑size asynchronous tasks and Group for known‑size batch tasks—offering a concise API, configurable workers, and built‑in callbacks to simplify high‑performance concurrent processing in backend applications.

Batch ProcessingLibrarygolang
0 likes · 9 min read
Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing
Python Crawling & Data Mining
Python Crawling & Data Mining
Jan 10, 2024 · Fundamentals

Batch Insert Images into Excel Cells with Python and openpyxl

This article demonstrates how to use Python's openpyxl library to automatically insert named images into specific Excel cells in bulk, explaining the problem, providing a complete script, and offering practical tips for handling large files and sharing the solution with the community.

Batch ProcessingExcel automationPython
0 likes · 5 min read
Batch Insert Images into Excel Cells with Python and openpyxl
FunTester
FunTester
Jan 9, 2024 · Databases

Boost Java Database Performance with PreparedStatement Batch Inserts

The article explains Java's PreparedStatement interface, highlighting its performance, security, and batch processing benefits, and provides a complete example of inserting multiple user records efficiently, while also discussing practical considerations such as batch size, transaction handling, error management, and optimization tips.

Batch ProcessingDatabase PerformanceJDBC
0 likes · 6 min read
Boost Java Database Performance with PreparedStatement Batch Inserts
DataFunTalk
DataFunTalk
Dec 22, 2023 · Big Data

Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto

This article details Li Auto's end‑to‑end data integration practice using Flink on Kubernetes, covering the evolution of their integration platform, architectural design, cloud‑native deployment, operational challenges, and future roadmap, while highlighting unified batch‑stream processing and resource elasticity.

Batch ProcessingBig DataCloud Native
0 likes · 12 min read
Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto
DataFunTalk
DataFunTalk
Dec 18, 2023 · Big Data

Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing

The article explains why unified data architecture is essential to avoid duplication and inefficiency, discusses differing performance trade‑offs among batch, streaming, and interactive analytics, introduces an incremental computation model that unifies these modes, and invites readers to a Dec 19, 2023 technical sharing event.

Batch ProcessingBig DataLambda Architecture
0 likes · 3 min read
Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing
dbaplus Community
dbaplus Community
Dec 14, 2023 · Big Data

How Flink Powers Unified Stream‑Batch Processing at Scale: Production Lessons

This article explains why Flink was chosen as a unified stream‑batch engine, details the migration from Lambda architecture, outlines the Flink Batch production workflow, and shares key optimizations such as Hive dialect support, CTAS, adaptive scheduling, speculative execution, and future roadmap for large‑scale data processing.

Adaptive SchedulerBatch ProcessingBig Data
0 likes · 31 min read
How Flink Powers Unified Stream‑Batch Processing at Scale: Production Lessons
vivo Internet Technology
vivo Internet Technology
Dec 13, 2023 · Big Data

Hudi Data Lake Implementation and Optimization Practice at vivo

Vivo’s big‑data team deployed Apache Hudi to create a lakehouse that unifies streaming and batch workloads, leverages COW and MOR storage modes, automates small‑file clustering and compaction, and applies extensive version, streaming, batch, and lifecycle optimizations, delivering minute‑level latency, hundred‑million‑records‑per‑minute ingestion, and query speeds up to 20 % faster than Hive.

Apache HudiBatch ProcessingBig Data
0 likes · 11 min read
Hudi Data Lake Implementation and Optimization Practice at vivo
Selected Java Interview Questions
Selected Java Interview Questions
Dec 12, 2023 · Backend Development

Spring Boot Integration with Spring Batch: A Complete Tutorial and Example

This article provides a step‑by‑step guide on integrating Spring Batch with Spring Boot, covering business scenarios, database setup, Maven dependencies, configuration of JobRepository, JobLauncher, Job, Step, ItemReader, ItemProcessor, ItemWriter, listeners, validators, execution via REST endpoints, troubleshooting with Druid and switching to HikariCP, and demonstrates processing CSV and database data in large batches.

Batch ProcessingMyBatisSpring Batch
0 likes · 19 min read
Spring Boot Integration with Spring Batch: A Complete Tutorial and Example
DaTaobao Tech
DaTaobao Tech
Dec 11, 2023 · Big Data

Design and Implementation of an Online Batch Processing Framework for Large-Scale Promotion Systems

The paper presents a centralized online batch‑processing framework for large‑scale promotion systems, where applications integrate via an SDK, a task‑center schedules and dispatches sub‑tasks through RocketMQ to Dubbo‑enabled containers, employing MapReduce‑style splitting, Guava rate‑limiting, heartbeat health checks, and has successfully handled over 1.3 million tasks during Double‑11.

Batch ProcessingBig DataDubbo
0 likes · 9 min read
Design and Implementation of an Online Batch Processing Framework for Large-Scale Promotion Systems
Java Architect Essentials
Java Architect Essentials
Dec 5, 2023 · Backend Development

Comprehensive Interface Performance Optimization Strategies

This article presents a collection of practical backend interface optimization techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, code restructuring, pagination, SQL tuning, and lock granularity—to reduce latency and improve overall system efficiency.

API optimizationAsynchronousBatch Processing
0 likes · 10 min read
Comprehensive Interface Performance Optimization Strategies
Architect
Architect
Nov 3, 2023 · Backend Development

How We Cut Risk Engine Latency from 80 ms to 25 ms with Prefetch, Batching, Async, Compression and Bloom‑Filter Caching

Facing a traffic surge that pushed a risk‑control engine's response time beyond 250 ms, the team applied a series of systematic optimizations—feature prefetching, batch requests, asynchronous accumulator updates, multi‑level caching with Bloom filters, and log‑compression redesign—resulting in latency dropping to 25 ms, CPU and memory usage falling by up to 90%, and storage costs reduced by over a third.

Batch ProcessingBloom filterasynchronous processing
0 likes · 22 min read
How We Cut Risk Engine Latency from 80 ms to 25 ms with Prefetch, Batching, Async, Compression and Bloom‑Filter Caching
Bilibili Tech
Bilibili Tech
Oct 25, 2023 · Backend Development

Performance Optimization Practices in Bilibili's Risk Control Engine

To overcome storage, compute, and I/O bottlenecks in Bilibili’s risk‑control engine, the team combined pre‑fetching with Redis caching, batch retrieval, asynchronous writes via Railgun, aggressive log compression, and a multi‑level cache plus Bloom filter, cutting latency to sub‑100 ms, reducing Redis QPS by over 90 % and storage by ~38 %, while supporting million‑level query throughput.

Batch ProcessingPrefetchasync
0 likes · 22 min read
Performance Optimization Practices in Bilibili's Risk Control Engine
Java Backend Technology
Java Backend Technology
Oct 9, 2023 · Backend Development

How to Merge Requests in Spring Boot to Reduce DB Load and Boost Performance

This article explains how to combine multiple user queries into a single database request using a queue, ScheduledThreadPoolExecutor, and CompletableFuture in Spring Boot, demonstrating code implementations, handling Java 8 CompletableFuture timeout limitations, and showing performance gains through request merging under high concurrency.

Batch ProcessingSpring Bootconcurrency
0 likes · 15 min read
How to Merge Requests in Spring Boot to Reduce DB Load and Boost Performance
Top Architect
Top Architect
Oct 8, 2023 · Backend Development

Merging Backend Requests in SpringBoot to Reduce Database Connections

This article explains how to merge multiple backend requests in a SpringBoot application using a blocking queue, ScheduledThreadPoolExecutor, and CompletableFuture to batch database queries, reduce connection overhead, handle high concurrency, and includes full Java code examples and performance testing.

Batch ProcessingJava concurrencySpringBoot
0 likes · 15 min read
Merging Backend Requests in SpringBoot to Reduce Database Connections
Architect
Architect
Oct 1, 2023 · Backend Development

Batch Request Merging in Spring Boot to Reduce Database Connections

This article demonstrates how to merge multiple user‑info requests on the server side using a blocking queue, ScheduledThreadPoolExecutor, and CompletableFuture in Spring Boot, thereby consolidating SQL queries into a single batch call to save database connection resources while handling high concurrency.

Batch ProcessingCompletableFutureJava concurrency
0 likes · 13 min read
Batch Request Merging in Spring Boot to Reduce Database Connections
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 18, 2023 · Big Data

Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse

The article explains the mainstream Lambda data‑warehouse architecture, its benefits and challenges, then introduces Hudi as a lake‑house solution that unifies real‑time and offline storage, describes the multi‑layer service design, and showcases three practical scenarios—stream processing, real‑time multidimensional analysis, and stream‑batch data reuse—demonstrating how the integrated architecture improves latency, cost, and operational complexity.

Batch ProcessingData WarehouseHudi
0 likes · 13 min read
Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse
Code Ape Tech Column
Code Ape Tech Column
Sep 16, 2023 · Backend Development

Batch Request Merging in Java to Reduce Database Connections

This article explains how to merge multiple user‑detail requests on the server side using a blocking queue, scheduled thread pool and CompletableFuture in Spring Boot, thereby converting many individual SQL calls into a single batch query, saving database connections and improving high‑concurrency performance.

Batch ProcessingCompletableFutureSpring Boot
0 likes · 13 min read
Batch Request Merging in Java to Reduce Database Connections
ITPUB
ITPUB
Sep 13, 2023 · Backend Development

Why Is Kafka So Fast? 7 Core Techniques Behind Its High Throughput

This article explains how Kafka achieves million‑message‑per‑second throughput by leveraging zero‑copy I/O, an append‑only log, batch processing, compression, consumer pull optimization, unflushed memory buffers, and JVM garbage‑collection tuning, detailing each mechanism and its impact on performance.

Batch ProcessingGC optimizationKafka
0 likes · 14 min read
Why Is Kafka So Fast? 7 Core Techniques Behind Its High Throughput
Architect's Tech Stack
Architect's Tech Stack
Sep 11, 2023 · Databases

Performance Evaluation of Large-Scale Data Insertion into MySQL Using MyBatis, JDBC, and Batch Processing

This article presents a systematic performance test of inserting massive data into MySQL, comparing three strategies—MyBatis lightweight insertion, direct JDBC (with and without transactions), and JDBC batch processing—showing how transaction handling and batch execution dramatically affect insertion speed.

Batch ProcessingJDBCMyBatis
0 likes · 15 min read
Performance Evaluation of Large-Scale Data Insertion into MySQL Using MyBatis, JDBC, and Batch Processing
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 21, 2023 · Big Data

Key Features and Benefits of Lakehouse Frameworks Hudi, Iceberg, and Paimon

This note outlines how Hudi, Iceberg, and Paimon provide unified batch‑stream storage, UPSERT support, time‑travel capabilities, and lower development costs, enabling a streaming‑warehouse architecture that offers near‑real‑time latency, consistent semantics, persisted intermediate results, and easier historical data repair.

Batch ProcessingHudiIceberg
0 likes · 5 min read
Key Features and Benefits of Lakehouse Frameworks Hudi, Iceberg, and Paimon
Architect
Architect
Jul 30, 2023 · Backend Development

Spring Boot Integration with Spring Batch: A Comprehensive Tutorial

This article provides a step‑by‑step guide on integrating Spring Batch with Spring Boot, covering configuration, job and step setup, ItemReader/Processor/Writer implementations, database interactions, handling CSV and database sources, and troubleshooting common issues such as connection‑pool compatibility.

Batch ProcessingSpring BatchSpring Boot
0 likes · 22 min read
Spring Boot Integration with Spring Batch: A Comprehensive Tutorial
Senior Tony
Senior Tony
Jul 29, 2023 · Fundamentals

Essential Coding Habits Every Engineer Should Master

The article outlines practical coding habits—thorough input validation, comprehensive logging, careful RPC handling, batch processing, cautious SQL execution, safe extensions, disciplined refactoring, minimal dependencies, data consistency, and avoiding over‑engineering—to help engineers write reliable, maintainable code.

Batch ProcessingData ConsistencyInput Validation
0 likes · 9 min read
Essential Coding Habits Every Engineer Should Master
Architect
Architect
Jul 10, 2023 · Big Data

Understanding Lambda Architecture for Real‑Time Billion‑Scale Data Analysis

This article explains the Lambda Architecture—a three‑layer big‑data processing model combining batch and speed layers to deliver accurate, low‑latency analytics, and illustrates its use with Twitter hashtag tracking and a smart‑parking recommendation system.

Batch ProcessingBig DataLambda Architecture
0 likes · 10 min read
Understanding Lambda Architecture for Real‑Time Billion‑Scale Data Analysis
DevOps
DevOps
Jul 4, 2023 · Operations

Best Practices for Batch Processing in Microservice Architecture

This article examines batch processing in microservice environments, defining its role versus data warehouses, exploring optimal placement across architectural layers, discussing access methods, and providing criteria such as complexity, consistency, security, performance, reliability, and scalability to guide the selection of an appropriate batch processing architecture.

Batch ProcessingData synchronizationMicroservices
0 likes · 11 min read
Best Practices for Batch Processing in Microservice Architecture
Efficient Ops
Efficient Ops
Jun 13, 2023 · Operations

How to Automate Batch Job Retries and Eliminate Midnight Outages

This article explores a real‑world scenario where a support manager faces nightly batch job interruptions, analyzes common database and environment failures, and presents a systematic redesign of the batch framework and executor to enable automatic retry, reducing manual intervention and improving operational reliability.

Batch ProcessingException HandlingMySQL
0 likes · 7 min read
How to Automate Batch Job Retries and Eliminate Midnight Outages
Architecture Breakthrough
Architecture Breakthrough
May 8, 2023 · Backend Development

Designing a Robust Batch Processing Module: Key Architecture Insights

This article outlines the essential architectural considerations for building a production‑ready batch processing module, covering design principles, task scheduling, parallelism, error handling, resource management, data‑layer concerns, deployment strategies, and monitoring practices.

Backend ArchitectureBatch ProcessingMonitoring
0 likes · 10 min read
Designing a Robust Batch Processing Module: Key Architecture Insights
DataFunSummit
DataFunSummit
Apr 28, 2023 · Big Data

Building a Unified Streaming‑Batch Storage Architecture at Xiaohongshu

This article presents Xiaohongshu's design and implementation of a unified streaming‑batch storage system that integrates Lambda architecture, Kafka, Flink, Iceberg, and modern OLAP engines to solve real‑time data warehouse pain points and enable consistent, exactly‑once analytics across streaming and batch workloads.

Batch ProcessingFlinkIceberg
0 likes · 16 min read
Building a Unified Streaming‑Batch Storage Architecture at Xiaohongshu
Java Architect Essentials
Java Architect Essentials
Apr 28, 2023 · Backend Development

12 Proven Strategies to Supercharge API Performance in Java

This article outlines twelve practical techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction sizing, code refactoring, pagination, SQL tuning, and lock granularity—to dramatically reduce API latency and improve overall backend efficiency.

API performanceAsynchronousBatch Processing
0 likes · 9 min read
12 Proven Strategies to Supercharge API Performance in Java
IT Architects Alliance
IT Architects Alliance
Apr 23, 2023 · Backend Development

Common Interface Performance Optimization Strategies

This article outlines a comprehensive set of backend interface optimization techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction sizing, code restructuring, deep pagination, SQL tuning, and lock granularity—to reduce latency and improve overall system efficiency.

API optimizationAsynchronous ExecutionBatch Processing
0 likes · 8 min read
Common Interface Performance Optimization Strategies
Top Architect
Top Architect
Apr 21, 2023 · Backend Development

Common Interface Performance Optimization Strategies

This article presents a comprehensive set of practical techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction size reduction, code refactoring, and pagination optimization—to systematically reduce API latency and improve overall backend service efficiency.

Batch ProcessingDatabasecaching
0 likes · 12 min read
Common Interface Performance Optimization Strategies
ByteDance Cloud Native
ByteDance Cloud Native
Apr 20, 2023 · Cloud Native

How Dragonfly Accelerates Image Distribution and Scales Kubernetes Batch Processing

At KubeCon+CloudNativeCon 2023 in Amsterdam, Volcano Engine and ByteDance presented two technical sessions covering Dragonfly's P2P image distribution best practices and large‑scale Kubernetes batch processing strategies, offering deep insights and real‑world implementations for cloud‑native developers.

Batch ProcessingDragonflyImage Distribution
0 likes · 4 min read
How Dragonfly Accelerates Image Distribution and Scales Kubernetes Batch Processing
ITPUB
ITPUB
Mar 24, 2023 · Big Data

What’s New in Apache Flink 1.17? Key Features, Performance Gains, and Streaming Warehouse Advances

Apache Flink 1.17 introduces a suite of batch and streaming enhancements—including a new Streaming Warehouse API, significant TPC‑DS performance boosts, adaptive batch scheduling, improved checkpointing, expanded SQL capabilities, Hive connector upgrades, and broader filesystem support—while also delivering upgrades to FRocksDB, Calcite, and the token framework to strengthen its position as a leading unified data‑processing engine.

Apache FlinkBatch ProcessingCheckpoint
0 likes · 23 min read
What’s New in Apache Flink 1.17? Key Features, Performance Gains, and Streaming Warehouse Advances
DataFunTalk
DataFunTalk
Mar 12, 2023 · Big Data

Apache Kyuubi 1.6.0 Feature Overview and Enhancements

The article provides a comprehensive walkthrough of Apache Kyuubi 1.6.0, detailing server‑side enhancements such as batch (JAR) task submission, metadata store and unified API/authentication, client‑side improvements to the built‑in JDBC driver and Beeline, as well as engine plugins for Spark, Flink, Trino and Hive, and concludes with the community’s roadmap and statistics.

Apache KyuubiBatch ProcessingBig Data
0 likes · 12 min read
Apache Kyuubi 1.6.0 Feature Overview and Enhancements
DataFunTalk
DataFunTalk
Feb 21, 2023 · Databases

Building a Stream‑Batch Integrated Data Architecture with Apache Doris at SelectDB

This article details how SelectDB’s data technology architect designed and implemented a new stream‑batch unified data platform using Apache Doris, covering the shortcomings of the early CDH‑based architecture, the selection process, data modeling, ingestion pipelines, performance testing, operational optimizations, and future plans.

Apache DorisBatch ProcessingBig Data
0 likes · 17 min read
Building a Stream‑Batch Integrated Data Architecture with Apache Doris at SelectDB
DataFunTalk
DataFunTalk
Feb 2, 2023 · Big Data

SeaTunnel: Design Goals, Current Status, Architecture, and Future Roadmap

This article provides a comprehensive overview of Apache SeaTunnel, covering its design objectives, current capabilities such as multi‑engine support and extensive connector ecosystem, detailed architecture including engine‑independent APIs and execution flows, and outlines the upcoming roadmap to expand connectors, launch a visual web UI, and introduce a dedicated SeaTunnel Engine.

Batch ProcessingBig DataConnector
0 likes · 12 min read
SeaTunnel: Design Goals, Current Status, Architecture, and Future Roadmap
DataFunSummit
DataFunSummit
Jan 8, 2023 · Big Data

Streaming‑Batch Integrated Real‑time Multi‑dimensional Analysis

This article presents a comprehensive overview of evolving big‑data architectures—from classic offline warehouses to Lambda and Kappa models—and details a streaming‑batch integrated solution that addresses latency, data freshness, and multi‑table join challenges to achieve minute‑level real‑time multi‑dimensional analytics.

Batch ProcessingData WarehouseKappa Architecture
0 likes · 18 min read
Streaming‑Batch Integrated Real‑time Multi‑dimensional Analysis
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 3, 2023 · Big Data

Migrating Hive SQL Jobs to Flink Using the SQL Gateway

This article explains how to use Apache Flink 1.16's SQL Gateway to migrate Hive SQL tasks to Flink, covering the underlying Hive‑on‑Flink architecture, dialect compatibility, streaming and batch demos, configuration details, and practical tips for developers and platform engineers.

Batch ProcessingBig DataFlink
0 likes · 19 min read
Migrating Hive SQL Jobs to Flink Using the SQL Gateway
Architects' Tech Alliance
Architects' Tech Alliance
Dec 25, 2022 · Operations

Mastering Time‑Space Trade‑offs: Indexing, Caching, Compression & More

This article consolidates three performance‑optimization posts, detailing six universal time‑space trade‑off techniques—indexing, caching, compression, prefetching, peak‑valley smoothing, and batch processing—plus four advanced parallelism strategies, illustrated with real‑world analogies and practical guidelines for developers.

Batch ProcessingIndexingPrefetch
0 likes · 22 min read
Mastering Time‑Space Trade‑offs: Indexing, Caching, Compression & More
Data Thinking Notes
Data Thinking Notes
Dec 21, 2022 · Big Data

Why Your Spark Batch Job Fails: Memory Limits, Data Skew, and Practical Fixes

This article examines a recurring Spark batch task failure caused by OutOfMemory errors and data skew, details the investigation steps—including increasing executor memory, raising parallelism, and analyzing shuffle metrics—and proposes solutions such as data validation, filtering oversized keys, and memory adjustments.

Batch ProcessingData SkewOutOfMemory
0 likes · 4 min read
Why Your Spark Batch Job Fails: Memory Limits, Data Skew, and Practical Fixes
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 15, 2022 · Big Data

Migrating Hive SQL to Flink SQL: Motivation, Challenges, Practice, Demo, and Future Plans

This technical article presents a comprehensive overview of migrating Hive SQL to Flink SQL, covering the motivations behind the migration, key challenges such as compatibility, stability and performance, practical implementation steps, a detailed demo, future development directions, and a Q&A session addressing common concerns.

Batch ProcessingBig DataData Lake
0 likes · 13 min read
Migrating Hive SQL to Flink SQL: Motivation, Challenges, Practice, Demo, and Future Plans
Top Architect
Top Architect
Dec 5, 2022 · Backend Development

Integrating Spring Batch with Spring Boot: Full Tutorial and Example

This article provides a comprehensive step‑by‑step guide on integrating Spring Batch with Spring Boot, covering configuration, entity and mapper definitions, job, step, reader, processor, writer implementations, database setup, error handling, and execution via a REST controller, complete with code examples.

Batch ProcessingMyBatisMySQL
0 likes · 22 min read
Integrating Spring Batch with Spring Boot: Full Tutorial and Example
Java High-Performance Architecture
Java High-Performance Architecture
Dec 4, 2022 · Backend Development

How to Efficiently Import and Export Millions of Records with EasyExcel and POI

This article explains the challenges of large‑scale Excel import/export in Java, compares POI workbook implementations, introduces EasyExcel for high‑performance handling of hundreds of thousands to millions of rows, and provides detailed code examples for batch querying, sheet management, and JDBC batch insertion with transaction control.

Batch ProcessingData ExportEasyExcel
0 likes · 23 min read
How to Efficiently Import and Export Millions of Records with EasyExcel and POI
Java Architect Essentials
Java Architect Essentials
Nov 24, 2022 · Backend Development

How to Efficiently Import and Export Millions of Rows with POI and EasyExcel

This article explains how to overcome the limitations of traditional Apache POI when handling massive Excel import/export tasks by comparing POI workbook types, selecting the appropriate one, and using EasyExcel together with batch database queries and JDBC transactions to process up to three million rows efficiently.

Apache POIBatch ProcessingEasyExcel
0 likes · 23 min read
How to Efficiently Import and Export Millions of Rows with POI and EasyExcel
Top Architect
Top Architect
Nov 22, 2022 · Big Data

Efficient Massive Excel Import/Export with POI and EasyExcel in Java

This article explains how to efficiently import and export massive datasets (up to millions of rows) between Excel and databases using Apache POI, SXSSF, and Alibaba's EasyExcel, comparing workbook types, outlining performance considerations, and providing Java code examples for batch processing, paging, and transaction management.

Batch ProcessingBig DataEasyExcel
0 likes · 23 min read
Efficient Massive Excel Import/Export with POI and EasyExcel in Java
ITPUB
ITPUB
Nov 18, 2022 · Big Data

How Xiaomi Uses Iceberg for Real‑Time Streaming and Batch Data Lakes

This article introduces Iceberg’s table‑format fundamentals, details Xiaomi’s large‑scale deployment of Iceberg for CDC and log ingestion, explores their streaming‑batch integration experiments, outlines future roadmap items, and provides a comprehensive Q&A covering practical challenges and solutions.

Batch ProcessingBig DataData Lake
0 likes · 23 min read
How Xiaomi Uses Iceberg for Real‑Time Streaming and Batch Data Lakes
Architecture Digest
Architecture Digest
Nov 7, 2022 · Backend Development

Comprehensive Guide to Software Performance Optimization: Indexing, Compression, Caching, Prefetching, Throttling, and Batch Processing

This article presents a thorough, multi‑part exploration of software performance optimization techniques—including indexing, compression, caching, prefetching, peak‑shaving, and batch processing—explaining their principles, trade‑offs, practical applications, and how they relate to hardware constraints and system design.

Batch ProcessingIndexingOptimization
0 likes · 36 min read
Comprehensive Guide to Software Performance Optimization: Indexing, Compression, Caching, Prefetching, Throttling, and Batch Processing
Top Architect
Top Architect
Oct 20, 2022 · Databases

Performance Evaluation of Inserting Billion‑Scale Data into MySQL Using Java: MyBatis vs JDBC vs Batch Processing

This article presents a detailed performance test of inserting massive data into MySQL with Java, comparing three strategies—MyBatis lightweight insertion without transactions, direct JDBC handling with and without transactions, and JDBC batch processing—showing timing results for millions of rows and offering practical recommendations for high‑throughput data loading.

Batch ProcessingJDBCMyBatis
0 likes · 14 min read
Performance Evaluation of Inserting Billion‑Scale Data into MySQL Using Java: MyBatis vs JDBC vs Batch Processing
DeWu Technology
DeWu Technology
Oct 10, 2022 · Big Data

Offline and Real-Time User Profile Fusion Architecture

The architecture combines a nightly batch job that generates offline user profiles stored in HBase with a Flink‑based stream layer that lazily loads those profiles on app start and creates real‑time updates, then fuses both streams into a unified, timestamp‑ordered profile in Redis, forming a Lambda‑style pipeline.

Batch ProcessingFlinkHBase
0 likes · 10 min read
Offline and Real-Time User Profile Fusion Architecture
Architect's Tech Stack
Architect's Tech Stack
Oct 9, 2022 · Backend Development

Spring Batch Overview: Architecture, Core Concepts, and Practical Usage

This article provides a comprehensive introduction to Spring Batch, covering its purpose for large‑scale data processing, overall architecture, key concepts such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip policies, and practical configuration examples with Java code.

Batch ProcessingSpring Batchjava
0 likes · 17 min read
Spring Batch Overview: Architecture, Core Concepts, and Practical Usage
Past Memory Big Data
Past Memory Big Data
Sep 8, 2022 · Backend Development

Implementing Fixed-Length Queues and Batch Consumption in Redis with Lua

The article details how a gaming data‑reporting pipeline uses Redis lists, sets and Lua scripts to build a pseudo‑message queue that provides per‑game message grouping, fixed‑length queues, and batch consumption while meeting strict timeliness and throughput requirements.

Batch ProcessingFixed-length queueLua
0 likes · 12 min read
Implementing Fixed-Length Queues and Batch Consumption in Redis with Lua
dbaplus Community
dbaplus Community
Sep 6, 2022 · Backend Development

How to Scale High‑Concurrency Write Operations with Asynchronous Patterns

This article examines a high‑traffic video‑watching scenario, identifies database write bottlenecks, and presents four asynchronous solutions—thread‑pool, in‑memory queue with scheduled tasks, message‑queue, and Agent‑plus‑MQ—detailing their implementation steps, trade‑offs, and best‑practice guidelines.

AsynchronousBatch ProcessingThread Pool
0 likes · 8 min read
How to Scale High‑Concurrency Write Operations with Asynchronous Patterns
DataFunTalk
DataFunTalk
Aug 25, 2022 · Big Data

Applying OpenMLDB for Efficient AI Toolchain and Data‑Driven Architecture at Akulaku

This article presents Akulaku’s practical experience with OpenMLDB, describing the company’s data‑driven requirements, the design of a unified stream‑batch architecture, implementation details across offline, online and RocksDB modes, and future recommendations for high‑performance, scenario‑agnostic big‑data processing.

AIBatch ProcessingOpenMLDB
0 likes · 17 min read
Applying OpenMLDB for Efficient AI Toolchain and Data‑Driven Architecture at Akulaku
DaTaobao Tech
DaTaobao Tech
Aug 11, 2022 · Big Data

Unify SQL Engine: Integrating Stream, Batch, and Online Computing for Data Warehousing

The article describes how fragmented real‑time, batch, and online data‑warehouse pipelines suffer from low productivity and inconsistent data quality, and introduces a unified SQL engine built on Apache Calcite that parses, optimizes, and compiles a single SQL statement into executable plans for ODPS, Flink, or Java, leveraging Janino code generation, multi‑backend state storage, and snapshot‑join semantics to boost performance and simplify development.

Batch ProcessingCalciteData Warehouse
0 likes · 16 min read
Unify SQL Engine: Integrating Stream, Batch, and Online Computing for Data Warehousing
Java High-Performance Architecture
Java High-Performance Architecture
Aug 1, 2022 · Backend Development

Master Spring Batch: From Basics to Advanced Architecture and Best Practices

This article introduces Spring Batch, a lightweight Java batch‑processing framework, explains its architecture and core concepts such as Job, Step, ItemReader/Writer/Processor, chunk handling, skip strategies, and provides practical guidelines for building reliable, high‑throughput batch jobs while avoiding common pitfalls like memory exhaustion.

Batch ProcessingChunkJob
0 likes · 18 min read
Master Spring Batch: From Basics to Advanced Architecture and Best Practices
Programmer DD
Programmer DD
Jul 22, 2022 · Backend Development

Master Spring Batch: From Core Concepts to Advanced Chunk Processing

This article provides a comprehensive introduction to Spring Batch, covering its purpose, core architecture, key concepts such as Job, Step, ItemReader/Processor/Writer, chunk processing, skip handling, best practices, and common pitfalls like memory exhaustion, all illustrated with code examples and diagrams.

Batch ProcessingChunkSkip Strategy
0 likes · 20 min read
Master Spring Batch: From Core Concepts to Advanced Chunk Processing
Java Architect Essentials
Java Architect Essentials
Jul 17, 2022 · Backend Development

Master Spring Batch: Core Concepts, Architecture, and Practical Tips

This article provides a comprehensive guide to Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, common pitfalls, and practical configuration examples for robust batch applications.

Batch ProcessingChunk ProcessingJob Scheduling
0 likes · 18 min read
Master Spring Batch: Core Concepts, Architecture, and Practical Tips
Code Ape Tech Column
Code Ape Tech Column
Jul 16, 2022 · Backend Development

Spring Batch Architecture Overview and Core Concepts

This article introduces Spring Batch as a lightweight, comprehensive batch‑processing framework for enterprise applications, explains its overall architecture, and details core concepts such as Job, JobInstance, JobParameters, JobExecution, Step, StepExecution, ExecutionContext, JobRepository, JobLauncher, ItemReader, ItemWriter, ItemProcessor, chunk processing, skip/failure handling, best‑practice guidelines, and common troubleshooting tips.

Batch ProcessingChunkItemReader
0 likes · 20 min read
Spring Batch Architecture Overview and Core Concepts
Java Backend Technology
Java Backend Technology
Jul 12, 2022 · Backend Development

Boost Java Batch Updates with Multithreading: A Practical Guide

This article explains how to efficiently handle large‑scale batch updates in Java by splitting the data set, using a thread pool for concurrent processing, and providing reusable utility code, all illustrated with a clear workflow diagram and step‑by‑step examples.

Batch ProcessingThread Poolbackend
0 likes · 7 min read
Boost Java Batch Updates with Multithreading: A Practical Guide
DataFunTalk
DataFunTalk
May 23, 2022 · Big Data

Real-Time Data Lake Practices at ByteDance: Architecture, Challenges, and Solutions

ByteDance shares its real‑time data lake implementation, covering the evolving definition of data lakes, six core capabilities, challenges such as data management, weak concurrent updates, performance, and log ingestion, and detailed solutions including Hudi Metastore Server, bucket indexing, multi‑stage use cases, and future roadmap.

Batch ProcessingHudiReal-time Data Lake
0 likes · 32 min read
Real-Time Data Lake Practices at ByteDance: Architecture, Challenges, and Solutions
Top Architect
Top Architect
May 15, 2022 · Backend Development

Understanding MyBatis Batch Processing: Common Pitfalls and Optimized Solutions

This article walks through the challenges of inserting massive data sets with MyBatis, explains the inner workings of batch execution, highlights common mistakes with commit, clearCache and flushStatements, and presents progressively refined code examples—including Oracle-specific optimizations—to dramatically improve performance.

Batch ProcessingMyBatisSQL
0 likes · 12 min read
Understanding MyBatis Batch Processing: Common Pitfalls and Optimized Solutions
IT Architects Alliance
IT Architects Alliance
May 14, 2022 · Backend Development

Request Collapsing Techniques: Hystrix Collapser, Custom BatchCollapser, and ConcurrentHashMultiset

This article compares three request‑collapsing techniques—Hystrix Collapser, a custom BatchCollapser, and Guava’s ConcurrentHashMultiset—detailing their designs, implementations, configurations, and suitable scenarios for reducing downstream load and improving system throughput, including code examples, timer‑based batching, and thread‑safe container usage.

Batch ProcessingGuavaHystrix
0 likes · 14 min read
Request Collapsing Techniques: Hystrix Collapser, Custom BatchCollapser, and ConcurrentHashMultiset
Java High-Performance Architecture
Java High-Performance Architecture
May 12, 2022 · Backend Development

Master MyBatis Batch Processing: Common Pitfalls and Optimized Solutions

This article walks readers through MyBatis batch processing from basics to advanced, highlighting common mistakes, explaining the roles of commit, clearCache, and flushStatements, and presenting three progressively refined code versions plus Oracle-specific optimizations to dramatically improve bulk insert performance.

Batch ProcessingMyBatisSQL
0 likes · 12 min read
Master MyBatis Batch Processing: Common Pitfalls and Optimized Solutions
Shopee Tech Team
Shopee Tech Team
Apr 28, 2022 · Big Data

Building Real-Time Data Warehouse with Flink + Hudi at Shopee

Shopee replaced its hourly Hive pipeline with a hybrid Flink‑Hudi real‑time data warehouse that groups Kafka topics, applies lightweight stream ETL, uses partial‑update MOR tables for multi‑stream joins and COW tables for versioned batches, cutting latency from about 90 minutes to 2–30 minutes and halving resource usage.

Apache FlinkApache HudiBatch Processing
0 likes · 20 min read
Building Real-Time Data Warehouse with Flink + Hudi at Shopee
Architecture Digest
Architecture Digest
Jan 21, 2022 · Big Data

Building a Real-Time Data Warehouse with Flink: Architecture, Core Concepts, and Practical Implementation

This article explains how to build a unified stream‑batch real‑time data warehouse using FlinkSQL, covering prerequisite knowledge, five core concepts, two implementation approaches, a comparison of traditional versus real‑time architectures, and a comprehensive hands‑on example, illustrated with diagrams.

Batch ProcessingFlinkReal-time Data Warehouse
0 likes · 6 min read
Building a Real-Time Data Warehouse with Flink: Architecture, Core Concepts, and Practical Implementation
Architecture Digest
Architecture Digest
Jan 20, 2022 · Backend Development

Implementing Scheduled Device Upgrade with Spring Batch and Quartz in Spring Boot

This article explains how to handle a PC‑triggered device upgrade record by using Quartz for timed execution and Spring Batch for bulk processing, detailing Maven dependencies, YAML configuration, service and batch classes, custom reader/writer logic, a processor that calls an upgrade‑dispatch API, and the overall challenges encountered.

Batch ProcessingQuartzSpring Batch
0 likes · 13 min read
Implementing Scheduled Device Upgrade with Spring Batch and Quartz in Spring Boot
Top Architect
Top Architect
Jan 17, 2022 · Backend Development

Implementing Device Upgrade Batch Processing with Spring Batch and Quartz

This article explains how to use Spring Batch together with Quartz to periodically process device upgrade records, covering Maven dependencies, configuration files, service implementation, batch job and step definitions, a custom processor for HTTP dispatch, entity mapping, and job retry scheduling for failed updates.

Batch ProcessingJob SchedulingQuartz
0 likes · 10 min read
Implementing Device Upgrade Batch Processing with Spring Batch and Quartz
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 9, 2021 · Big Data

Apache Flink 1.7–1.14 Release Highlights and Feature Evolution

This article provides a comprehensive overview of Apache Flink's major releases from version 1.7 to 1.14, detailing new APIs, state management improvements, Kubernetes integration, SQL and Table API enhancements, checkpointing advances, and performance optimizations that together illustrate the platform's evolution for both streaming and batch processing workloads.

Apache FlinkBatch ProcessingCheckpoint
0 likes · 78 min read
Apache Flink 1.7–1.14 Release Highlights and Feature Evolution
MaGe Linux Operations
MaGe Linux Operations
Sep 30, 2021 · Operations

Boost Your Productivity: Automate Excel with Python and xlwings

This guide demonstrates how to use Python's xlwings library to programmatically create, rename, list, print, and split Excel workbooks and worksheets, enabling batch processing and automation that significantly improves efficiency for repetitive spreadsheet tasks.

Batch ProcessingExcel automationScripting
0 likes · 5 min read
Boost Your Productivity: Automate Excel with Python and xlwings
Big Data Technology Architecture
Big Data Technology Architecture
Aug 10, 2021 · Big Data

Building a Real‑Time Data Warehouse with Apache Flink and Apache Iceberg: Architecture, Challenges, and Best Practices

This article presents Tencent's practical experience of constructing a real‑time data warehouse by integrating Apache Flink with Apache Iceberg, covering background pain points of traditional Lambda architectures, Iceberg's table format and capabilities, Flink‑Iceberg sink design, small‑file handling, and future roadmap for a unified streaming‑batch data lake.

Apache FlinkApache IcebergBatch Processing
0 likes · 20 min read
Building a Real‑Time Data Warehouse with Apache Flink and Apache Iceberg: Architecture, Challenges, and Best Practices
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 30, 2021 · Big Data

Enterprise Big Data Platform Architecture: Insights from Taobao, Meituan, and Didi

This article examines the architecture of enterprise-level big data platforms at leading Chinese tech firms—Taobao, Meituan, and Didi—detailing their data sources, synchronization components, batch and streaming processing layers, scheduling systems, and common design patterns, while highlighting shared principles across these implementations.

Batch ProcessingStreamingarchitecture
0 likes · 9 min read
Enterprise Big Data Platform Architecture: Insights from Taobao, Meituan, and Didi
macrozheng
macrozheng
Jul 28, 2021 · Backend Development

Master Spring Batch: Core Concepts, Architecture, and Best Practices

This article provides a comprehensive overview of Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, execution contexts, chunk processing, skip strategies, and practical tips for configuration and memory management.

Batch ProcessingChunk ProcessingJob Scheduling
0 likes · 20 min read
Master Spring Batch: Core Concepts, Architecture, and Best Practices
Liangxu Linux
Liangxu Linux
Jul 18, 2021 · Backend Development

Master Bulk File Renaming with F2: Install, Commands, and Tips

F2 is a fast, safe Go‑based command‑line tool for batch renaming files and folders on Linux, macOS, and Windows, offering multiple installation methods, preview mode, undo, conflict resolution, built‑in variables, and powerful regex‑driven rename examples.

Batch ProcessingCLIFile Renaming
0 likes · 7 min read
Master Bulk File Renaming with F2: Install, Commands, and Tips
Architecture Digest
Architecture Digest
Jul 17, 2021 · Backend Development

Introduction to Spring Batch and Its Core Concepts

Spring Batch is a lightweight, comprehensive Java batch processing framework that provides reusable features such as job/step architecture, ItemReader/Writer/Processor, chunk processing, transaction management, and restart capabilities, with detailed explanations of core concepts, configuration examples, and best practices for building robust enterprise batch jobs.

Batch ProcessingChunkJob
0 likes · 19 min read
Introduction to Spring Batch and Its Core Concepts
Top Architect
Top Architect
Jul 13, 2021 · Backend Development

Introduction to Spring Batch and Its Core Concepts

This article provides a comprehensive overview of Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, and practical guidelines for building robust batch processing solutions in Java.

Batch ProcessingData ProcessingSpring Batch
0 likes · 19 min read
Introduction to Spring Batch and Its Core Concepts
Python Programming Learning Circle
Python Programming Learning Circle
Jul 8, 2021 · Operations

Python Office Automation Techniques: Document Conversion, Geocoding, Distance Calculation, Batch File Processing, and Outlook Email Automation

This article presents a collection of Python scripts for office automation, covering Word doc‑to‑docx conversion, batch address geocoding using Baidu API, distance computation with geopy, coordinate conversion between Baidu and Gaode, Excel merging, Word‑to‑PDF conversion, extracting tables from Word, and bulk Outlook email sending.

Batch ProcessingFile ConversionGeocoding
0 likes · 11 min read
Python Office Automation Techniques: Document Conversion, Geocoding, Distance Calculation, Batch File Processing, and Outlook Email Automation