BestHub
Sign in
Tagged articles

Batch Processing

260 articles · Page 1 of 3
LuTiao Programming
LuTiao Programming
Sep 30, 2026 · Backend Development

Spring Boot Excel Export OOM: Replacing XSSFWorkbook with SXSSFWorkbook and Batch Processing

The author describes how exporting 500k order records caused a 2GB heap OOM, then fixed it by replacing XSSFWorkbook with streaming SXSSFWorkbook, switching from full entity loading to cursor-based batch queries with DTOs, reusing cell styles, and finally decoupling large exports from HTTP requests via async task processing.

Apache POIAsync ProcessingBatch Processing
0 likes · 14 min read
Spring Boot Excel Export OOM: Replacing XSSFWorkbook with SXSSFWorkbook and Batch Processing
DataFunTalk
DataFunTalk
Sep 21, 2026 · Artificial Intelligence

Jev's First Week: 15 Projects Show How Agents Separate Generation from Judgment

In Jev's first six days, 239 public builds demonstrate a shift where developers offload structured judgments — yes/no, choice, scoring — to a specialized model while reserving LLMs for generation and code for execution, across browser automation, coding agents, batch classification, content scoring, and SQL integration.

Agent ArchitectureBatch ProcessingBrowser Agent
0 likes · 25 min read
Jev's First Week: 15 Projects Show How Agents Separate Generation from Judgment
Xiaolin Talks Programming
Xiaolin Talks Programming
Sep 18, 2026 · Backend Development

Engineering Large-Scale Excel Imports: Spring Boot + EasyExcel Streaming & Task Isolation

This article details a production-ready engineering approach for importing hundreds of thousands of Excel rows using Spring Boot and EasyExcel, covering streaming parsing, batch validation, async task isolation, idempotency, error reporting, and operational concerns like logging and temp file cleanup.

Async ProcessingBatch ProcessingData Validation
0 likes · 23 min read
Engineering Large-Scale Excel Imports: Spring Boot + EasyExcel Streaming & Task Isolation
IT Services Circle
IT Services Circle
Sep 15, 2026 · Backend Development

How to Diagnose and Resolve CPU/Memory Spikes in Agent Batch Processing

This article details a systematic approach to diagnosing and resolving CPU and memory spikes when an Agent processes large document batches, covering backpressure propagation, concurrency budgeting, runtime profiling, and pipeline design to prevent OOM kills and throttling.

AgentBackpressureBatch Processing
0 likes · 23 min read
How to Diagnose and Resolve CPU/Memory Spikes in Agent Batch Processing
Architect's Guide
Architect's Guide
Aug 26, 2026 · Backend Development

10 Powerful Performance‑Optimization Techniques You Should Try

The article surveys ten practical performance‑optimization tactics—from classic indexing, compression, and caching to prefetching, peak‑shaving, batch processing, and advanced methods such as resource squeezing, horizontal scaling, sharding, and lock‑free designs—explaining their trade‑offs, concrete examples, and when to apply each in real‑world systems.

Batch ProcessingCachingCompression
0 likes · 36 min read
10 Powerful Performance‑Optimization Techniques You Should Try
Xike
Xike
Aug 3, 2026 · Big Data

Turn Daily Batches into Re‑runnable Workflows with DolphinScheduler (Big Data Series 8)

This article walks through using Apache DolphinScheduler 3.2.2 in a lightweight Docker standalone setup to model, create, and run an ODS → ETL → ADS offline workflow via the Open API, demonstrating dependency handling, retries, backfill, and troubleshooting for reliable daily batch processing.

BackfillBatch ProcessingDolphinScheduler
0 likes · 12 min read
Turn Daily Batches into Re‑runnable Workflows with DolphinScheduler (Big Data Series 8)
Linyb Geek Road
Linyb Geek Road
Aug 1, 2026 · Artificial Intelligence

Practical Guide to Cutting LLM Token Costs

This article systematically explains how large‑language‑model token pricing works, identifies eight high‑consumption usage patterns, presents nine actionable optimization principles, and offers a tiered model‑selection framework so engineering teams can reduce token spend by up to 80% without sacrificing result quality.

Batch ProcessingCachingLLM
0 likes · 22 min read
Practical Guide to Cutting LLM Token Costs
YiSu Grain
YiSu Grain
Jul 28, 2026 · Big Data

Day 39: Big Data Architecture – Distributed Storage, Batch vs Stream Processing, and Compute‑Storage Separation

This lesson explains why a single database cannot scale for massive e‑commerce data, introduces the three core pillars of distributed storage—sharding, replication, and horizontal scaling—covers batch and stream processing differences with Spark, Hive, Flink and Storm, compares compute‑storage integration versus separation, and shows how Kafka, HDFS, Spark and Flink fit together in a real‑time data platform.

Batch ProcessingCompute-Storage SeparationFlink
0 likes · 25 min read
Day 39: Big Data Architecture – Distributed Storage, Batch vs Stream Processing, and Compute‑Storage Separation
CTO Full-Stack Academy
CTO Full-Stack Academy
Jul 28, 2026 · Backend Development

Mastering Batch Processing: Core Principles, Pitfalls, and Real‑World Cases

This article explains the fundamentals of batch processing, outlines its non‑real‑time nature, lists typical scenarios and pros‑cons, provides practical implementation guidelines, and presents three detailed industry case studies illustrating how large‑scale offline jobs are built and maintained.

BackendBatch ProcessingSystem Design
0 likes · 11 min read
Mastering Batch Processing: Core Principles, Pitfalls, and Real‑World Cases
Subtle Storm
Subtle Storm
Jul 19, 2026 · Big Data

Core Big Data Technologies Explained with a Restaurant Analogy

The article breaks down the essential big data components—data ingestion, distributed storage, resource management, batch and stream processing, analytics, search, and cluster operations—using a restaurant metaphor and lists typical tools such as Flume, Kafka, HDFS, Spark, Flink, Elasticsearch, and Kubernetes.

Batch ProcessingData AnalyticsData Ingestion
0 likes · 7 min read
Core Big Data Technologies Explained with a Restaurant Analogy
Random Bulletin
Random Bulletin
Jul 17, 2026 · Backend Development

Why Moving from Real‑Time to Batch Is Essential for Scaling to Tens of Millions QPS

Scaling a service from millions to tens of millions of queries per second fails not because of data size but due to per‑request fixed costs, and the article shows how batching aggregates these costs, dramatically boosts throughput, reduces latency, and introduces new challenges such as memory pressure and partial failures.

Batch ProcessingSystem Designhigh QPS
0 likes · 17 min read
Why Moving from Real‑Time to Batch Is Essential for Scaling to Tens of Millions QPS
samdeepthink
samdeepthink
Jul 16, 2026 · Databases

Checkpoint‑Based Resumption for Billions‑Row Full Migration

When migrating tens of billions of rows, the article explains how a simple progress‑tracking table and failure‑log table enable automatic checkpoint‑based resumption, stateless execution, and dynamic batch tuning without restarting or rewriting code.

Batch ProcessingCheckpointMySQL
0 likes · 5 min read
Checkpoint‑Based Resumption for Billions‑Row Full Migration
Spring Full-Stack Practical Cases
Spring Full-Stack Practical Cases
Jul 11, 2026 · Backend Development

Boost Performance in Spring Boot with a Single Batch‑Processing Annotation

The article demonstrates how to create a custom @BatchProcess annotation combined with AOP to aggregate high‑frequency requests into batches, persist metadata in Redis, and process them efficiently, thereby reducing connection, I/O, and CPU overhead in distributed, high‑concurrency Spring Boot 3.5.0 applications.

AOPBatch ProcessingCustom Annotation
0 likes · 14 min read
Boost Performance in Spring Boot with a Single Batch‑Processing Annotation
Xiaolin Talks Programming
Xiaolin Talks Programming
Jul 7, 2026 · Backend Development

Spring Batch for Massive Data Migration: Partitioning, Fault Tolerance & Checkpoint Restart

This article details production-ready Spring Batch architecture for massive data migration, covering core Job/Step/Chunk model, partitioning for parallelism, streaming file reads, JDBC batch tuning, retry/skip fault tolerance, checkpoint restart via BATCH_* tables, memory/cursor trade-offs, dynamic chunk sizing, monitoring metrics, and scheduler-executor decoupling with XXL-JOB or Airflow.

Batch ProcessingCheckpoint RestartJDBC Optimization
0 likes · 19 min read
Spring Batch for Massive Data Migration: Partitioning, Fault Tolerance & Checkpoint Restart
Cloud Architecture
Cloud Architecture
Jul 6, 2026 · Backend Development

Designing a High‑Concurrency Coupon Expiration System with Task Tables and Batch Processing

The article explains why coupon expiration cannot be handled by a simple scheduled scan and presents a production‑grade architecture that uses a task‑table, four‑plane design, fine‑grained splitting, lease‑based worker coordination, idempotent updates, and observability to reliably expire billions of coupons under peak load.

Batch ProcessingConcurrencyMySQL
0 likes · 37 min read
Designing a High‑Concurrency Coupon Expiration System with Task Tables and Batch Processing
Advanced AI Application Practice
Advanced AI Application Practice
Jul 1, 2026 · Operations

Still Copy‑Pasting Bugs? This Skill Inserts Bugs into Feishu Base in 10 Seconds

The bug‑report‑writer v1.1.1 upgrade adds Feishu Base one‑click API writing, cutting single‑bug entry time from two minutes to 10‑15 seconds, boosting overall bug‑submission efficiency by 80% over v1.0 and up to 72‑fold versus manual entry, while supporting batch imports and a six‑dimension quality score.

Batch ProcessingFeishuautomation
0 likes · 11 min read
Still Copy‑Pasting Bugs? This Skill Inserts Bugs into Feishu Base in 10 Seconds
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 28, 2026 · Artificial Intelligence

DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights

DSpark, DeepSeek’s new LLM inference framework, combines batch processing, speculative decoding, Eagle‑style draft models and DFlash‑style parallel generation with a lightweight sequential head and hardware‑aware scheduling, delivering 60‑85% speedups while preserving model quality.

Batch ProcessingDeepSeekGPU optimization
0 likes · 12 min read
DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights
Niu Liu
Niu Liu
Jun 25, 2026 · Backend Development

Don’t Mix Them Up: When to Use RPC, MQ, or Offline Scripts

The article breaks down three fundamental mechanisms—synchronous RPC, asynchronous MQ, and offline scripts—by comparing their lifecycles, IO models, bottlenecks, and consistency guarantees, and provides a concrete decision framework for choosing the right tool for different business scenarios.

Batch ProcessingRPCconsistency
0 likes · 12 min read
Don’t Mix Them Up: When to Use RPC, MQ, or Offline Scripts
Shepherd Advanced Notes
Shepherd Advanced Notes
Jun 24, 2026 · Backend Development

Boosting Throughput 10×: Architecture Evolution and Tuning for High‑Concurrency Batch Processing

The article details how a sluggish batch‑processing system handling millions of records was redesigned with XXL‑JOB sharding, Redis‑based dynamic task distribution, cursor pagination, and selective transaction scopes, achieving nearly ten‑fold throughput improvement while addressing resource contention, load‑balancing, and reliable result reconciliation.

Batch ProcessingJavaMySQL
0 likes · 19 min read
Boosting Throughput 10×: Architecture Evolution and Tuning for High‑Concurrency Batch Processing
Java Tech Workshop
Java Tech Workshop
Jun 9, 2026 · Backend Development

Spring Batch Basics: Building Efficient SpringBoot Batch Jobs

This article explains why naive for‑loop DB operations fail on large data sets, introduces Spring Batch’s chunk, transaction, retry, skip and restart features, and provides step‑by‑step SpringBoot configurations, code samples for Tasklet and Chunk jobs, database and CSV readers/writers, and manual or scheduled job triggering.

Batch ProcessingChunkRetry
0 likes · 18 min read
Spring Batch Basics: Building Efficient SpringBoot Batch Jobs
Cloud Architecture
Cloud Architecture
May 30, 2026 · Operations

How to Build Production‑Grade Observability Metrics and Alerting for Batch Jobs

The article explains why batch processing tasks often slip out of control, defines a four‑layer observability model covering status, progress, quality and performance, proposes a unified task state machine and event flow, and provides concrete metric, logging, tracing and alerting designs—including Go and Java SDK examples—for reliable production‑level batch job monitoring.

Batch ProcessingGoJava
0 likes · 34 min read
How to Build Production‑Grade Observability Metrics and Alerting for Batch Jobs
Cloud Architecture
Cloud Architecture
May 29, 2026 · Cloud Native

Batch Task Platform on Kubernetes: From Job Wrappers to Scalable Control Plane

The article explains how to design a production‑grade, unified batch‑task platform on Kubernetes that goes beyond a simple job UI, covering unified abstractions, multi‑tenant governance, state‑machine modeling, scalable scheduling, high‑concurrency handling, observability, security, and a phased roadmap for incremental implementation.

Batch ProcessingKubernetescloud-native
0 likes · 36 min read
Batch Task Platform on Kubernetes: From Job Wrappers to Scalable Control Plane
Cloud Architecture
Cloud Architecture
May 29, 2026 · Cloud Native

DAG‑as‑Code: Building a Production‑Grade Batch Orchestration System with Argo Workflows

This article explains how to design and operate a scalable, multi‑tenant batch processing platform on Kubernetes using Argo Workflows, covering core concepts, DAG scheduling, concurrency control, controller scaling, artifact handling, event‑driven triggers, and practical best‑practice patterns for production reliability and cost efficiency.

Argo WorkflowsBatch ProcessingKubernetes
0 likes · 33 min read
DAG‑as‑Code: Building a Production‑Grade Batch Orchestration System with Argo Workflows
Cloud Architecture
Cloud Architecture
May 28, 2026 · Cloud Native

Production‑Ready Guide to Kubernetes Jobs & CronJobs: Controller Mechanics to Batch Platform Design

This article explains how Kubernetes Jobs and CronJobs work under the hood, outlines production‑grade design principles such as idempotency, failure modeling, scaling, observability, and security, and provides concrete YAML configurations and Go code examples for building a reliable, high‑throughput batch processing platform.

Batch ProcessingCronJobGo
0 likes · 46 min read
Production‑Ready Guide to Kubernetes Jobs & CronJobs: Controller Mechanics to Batch Platform Design
Cloud Architecture
Cloud Architecture
Apr 27, 2026 · Backend Development

Building an Enterprise‑Level MyBatis Persistence Layer from Zero to One

The article walks through a real production incident caused by a massive IN‑list query, then presents a complete methodology for designing, implementing, and tuning an enterprise‑grade MyBatis persistence layer—including core execution chain, caching strategies, batch processing, read/write splitting, sharding, observability, and deployment best practices.

Batch ProcessingCachingMyBatis
0 likes · 39 min read
Building an Enterprise‑Level MyBatis Persistence Layer from Zero to One
Architect's Guide
Architect's Guide
Apr 27, 2026 · Backend Development

How to Optimize Large Transactions in API Endpoints

The article examines why large database transactions inside API endpoints degrade performance, outlines issues such as data inconsistency, lock contention, undo‑log overhead, and DB pressure, and presents practical optimizations including removing remote RPC calls, using programmatic transactions, batch processing, splitting into smaller transactions, and asynchronous parallel execution with code examples.

Batch ProcessingProgrammatic TransactionTransactions
0 likes · 9 min read
How to Optimize Large Transactions in API Endpoints
JD Tech
JD Tech
Apr 16, 2026 · Industry Insights

How JD Revolutionized Coupon Search with a Stream‑Batch Unified Architecture

This article analyzes JD's end‑to‑end upgrade of its retail coupon search infrastructure, detailing the business drivers, data‑skew challenges, the shift from dual KV and batch pipelines to a unified stream‑batch model built on Apache Doris, and the resulting performance, resource and stability gains across multiple scenarios.

Apache DorisBatch ProcessingCoupon Search
0 likes · 12 min read
How JD Revolutionized Coupon Search with a Stream‑Batch Unified Architecture
AI Explorer
AI Explorer
Mar 16, 2026 · Artificial Intelligence

How to Unlock NotebookLM’s Python API for Batch Automation and AI Workflows

The unofficial notebooklm-py library provides a Python API and CLI that give programmatic access to Google NotebookLM’s hidden capabilities, enabling batch import of PDFs or videos, automated generation of podcasts, mind‑maps, and other outputs, and integration into AI agents for scalable research workflows.

AI AgentsAI automationBatch Processing
0 likes · 6 min read
How to Unlock NotebookLM’s Python API for Batch Automation and AI Workflows
LuTiao Programming
LuTiao Programming
Mar 14, 2026 · Backend Development

Why Your Spring Boot App Freezes at One Million Records – 5 Proven Techniques to Double Performance

When a Spring Boot application reaches millions of rows, it often suffers from OutOfMemoryErrors, slow queries, and high CPU, but by applying five proven strategies—pagination, streaming, batch processing, indexing, and asynchronous execution—you can halve memory usage and achieve up to ten‑fold speed gains.

Asynchronous ExecutionBatch ProcessingIndexing
0 likes · 11 min read
Why Your Spring Boot App Freezes at One Million Records – 5 Proven Techniques to Double Performance
Subtle Storm
Subtle Storm
Feb 13, 2026 · Artificial Intelligence

Using Ollama API for Custom Model Parameters and Batch Processing in Python

This article demonstrates how to configure advanced Ollama API options—including temperature, top‑p, and custom model creation—and shows a complete Python batch‑processing workflow that benchmarks multiple LLMs while measuring latency and success rates.

APIBatch ProcessingCustom Model
0 likes · 6 min read
Using Ollama API for Custom Model Parameters and Batch Processing in Python
Lakehouse Research Base
Lakehouse Research Base
Feb 12, 2026 · Big Data

Spark & Flink in Lakehouse: Latency, Accuracy & Cost Trade-offs for Batch-Stream Design

This article analyzes how Spark and Flink should divide responsibilities in lakehouse architectures by evaluating three core dimensions—data latency, computation accuracy, and resource cost—arguing that mature batch-stream integration balances business needs against technical overhead rather than pursuing universal real-time processing.

Batch ProcessingComputation AccuracyData Latency
0 likes · 9 min read
Spark & Flink in Lakehouse: Latency, Accuracy & Cost Trade-offs for Batch-Stream Design
Architect's Guide
Architect's Guide
Jan 16, 2026 · Databases

How to Safely Update Billions of MySQL Rows Without Overloading Binlog

This article explains why a naïve full‑table UPDATE on massive MySQL tables can cripple replication, explores deep‑pagination and IN‑clause inefficiencies, and presents a batch‑processing strategy using NO_CACHE, FORCE INDEX, and rate‑controlled scripts to perform safe, high‑performance updates.

Batch ProcessingBinlogFull Table Update
0 likes · 8 min read
How to Safely Update Billions of MySQL Rows Without Overloading Binlog
SpringMeng
SpringMeng
Jan 8, 2026 · Backend Development

How to Cut API Latency from Seconds to Milliseconds: Elegant Optimization Techniques

This article walks through a series of practical backend techniques—batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction management, pagination, and lock granularity—to dramatically reduce API response times from several seconds to just a few milliseconds.

API optimizationBatch ProcessingCaching
0 likes · 9 min read
How to Cut API Latency from Seconds to Milliseconds: Elegant Optimization Techniques
DeWu Technology
DeWu Technology
Dec 25, 2025 · Frontend Development

Build a High‑Performance H5 PAG Player: SDK, Image Editing, Batch Synthesis

This guide details how to implement a full‑stack H5 PAG player for the “Use Basketball to Know Me” activity, covering SDK loading, canvas‑based image manipulation (drag, scale, rotate), dynamic layer and text replacement, real‑time preview synchronization, snapshot export, batch synthesis, performance tuning, and fallback strategies.

Batch ProcessingCanvasPAG
0 likes · 30 min read
Build a High‑Performance H5 PAG Player: SDK, Image Editing, Batch Synthesis
Architect's Tech Stack
Architect's Tech Stack
Dec 24, 2025 · Backend Development

10 Proven Techniques to Supercharge API Performance in Java Applications

This article presents a comprehensive, step‑by‑step guide to optimizing Java‑based API interfaces, covering batch processing, asynchronous execution, caching, pre‑processing, pooling, transaction handling, pagination, SQL tuning, lock granularity, and code restructuring, with practical code examples and diagrams.

API optimizationBatch ProcessingCaching
0 likes · 10 min read
10 Proven Techniques to Supercharge API Performance in Java Applications
FunTester
FunTester
Dec 23, 2025 · Backend Development

Mastering Delayed, Priority, and Retry Tasks with River – A Go Queue Deep Dive

This article explains how River, a Go job‑queue library, implements delayed execution, priority handling, exponential‑backoff retries, batch inserts, monitoring, and best‑practice patterns, and compares it with other queue solutions to help developers build reliable, high‑performance background processing pipelines.

Batch ProcessingGoRetry Mechanism
0 likes · 14 min read
Mastering Delayed, Priority, and Retry Tasks with River – A Go Queue Deep Dive
Ray's Galactic Tech
Ray's Galactic Tech
Dec 12, 2025 · Cloud Native

Mastering Kubernetes Jobs and CronJobs: Complete Guide & Practical Examples

Learn how Kubernetes Jobs and CronJobs enable one‑off and scheduled batch processing, understand their core concepts, key differences, YAML specifications, typical use cases, advanced configurations, monitoring, logging, and cleanup strategies, and see real‑world examples with complete YAML snippets and command‑line tips.

Batch ProcessingCronJobJob
0 likes · 8 min read
Mastering Kubernetes Jobs and CronJobs: Complete Guide & Practical Examples
Chen Tian Universe
Chen Tian Universe
Nov 27, 2025 · Fundamentals

A Comprehensive Overview of Modern Bank Core Systems Architecture

This article provides a detailed walkthrough of bank core system architecture, covering its evolution, global overview, key modules such as customer information, account types, deposit and loan cores, payment and settlement, general ledger, card management, and batch processing, illustrated with diagrams and real‑world examples.

Account ManagementBatch ProcessingLedger
0 likes · 20 min read
A Comprehensive Overview of Modern Bank Core Systems Architecture
Tech Freedom Circle
Tech Freedom Circle
Nov 16, 2025 · Databases

How Redis Pipeline Can Boost Performance 3‑12× and Impress Interviewers

This article explains Redis Pipeline’s core principle of batching commands to reduce network round‑trips, presents benchmark data showing up to 17‑fold speedups, details real‑world use cases such as cache warm‑up, heartbeat reporting, and high‑traffic events, and provides best‑practice guidelines on batch sizing, error handling, cluster constraints, and comparisons with transactions and Lua scripts.

Batch ProcessingBenchmarkJava
0 likes · 36 min read
How Redis Pipeline Can Boost Performance 3‑12× and Impress Interviewers
dbaplus Community
dbaplus Community
Nov 12, 2025 · Databases

Mastering Data Sync: From Full Loads to Real‑Time CDC in E‑Commerce

This guide walks a new e‑commerce developer through the evolution of order data synchronization—from naïve full‑table loads, through incremental and batch strategies, cursor‑based pagination, performance tuning, and finally to real‑time CDC with message queues—highlighting pitfalls and practical solutions.

Batch ProcessingCDCCursor Pagination
0 likes · 12 min read
Mastering Data Sync: From Full Loads to Real‑Time CDC in E‑Commerce
Architect-Kip
Architect-Kip
Nov 4, 2025 · Backend Development

How to Optimize Batch Jobs for Better System and Database Performance

This article explains why batch tasks significantly affect system and database performance, and outlines three key strategies—task simplification, resource and performance tuning, and proactive monitoring with fault recovery—to design and optimize batch jobs effectively in enterprise development.

Batch ProcessingResource Isolationdatabase
0 likes · 4 min read
How to Optimize Batch Jobs for Better System and Database Performance
Su San Talks Tech
Su San Talks Tech
Oct 29, 2025 · Databases

How to Safely Delete Millions of Rows from a Large Table: Proven Strategies & Code

This article explains why deleting tens of millions of rows is challenging, analyzes transaction, lock, resource, and business impacts, and presents five practical solutions—including batch deletion, table rebuild, partition dropping, online migration, and professional tools—along with best‑practice guidelines and a decision matrix.

Batch ProcessingMySQLPartitioning
0 likes · 28 min read
How to Safely Delete Millions of Rows from a Large Table: Proven Strategies & Code
Architect-Kip
Architect-Kip
Oct 28, 2025 · Operations

Mastering Failure Recovery: Fast‑Fail, Auto‑Retry, and Resilience Patterns for Distributed Systems

This guide outlines core principles and practical solutions for building resilient backend systems, covering fast‑failure handling, automatic retries with exponential back‑off, circuit‑breaker usage, idempotency, batch job strategies, online transaction patterns, and robust message‑queue processing.

Batch ProcessingCircuit BreakerFailure Recovery
0 likes · 17 min read
Mastering Failure Recovery: Fast‑Fail, Auto‑Retry, and Resilience Patterns for Distributed Systems
Top Architect
Top Architect
Oct 28, 2025 · Backend Development

How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s

This article details a step‑by‑step performance overhaul for importing 60,000+ XML records into MySQL, covering baseline measurements, MySQL batch processing, asynchronous writes with Disruptor, XML parsing optimizations, and tuning MySQL buffers, ultimately reducing total runtime from 300 seconds to just four seconds.

Batch ProcessingDisruptorJava
0 likes · 14 min read
How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s
Chen Tian Universe
Chen Tian Universe
Oct 14, 2025 · Operations

How Banks Turn Every Transaction into Precise Accounting Records

This article explains the core banking accounting system, covering the transaction‑driven entry engine, multidimensional accounting, internal account management, end‑of‑day batch processing, and a complete loan lifecycle example that shows how each step is recorded, reconciled, and reported.

Batch Processingbankingcore accounting
0 likes · 16 min read
How Banks Turn Every Transaction into Precise Accounting Records
Architect's Guide
Architect's Guide
Oct 3, 2025 · Backend Development

Merge Duplicate Requests Using Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset

This article explains how merging similar or duplicate requests upstream with Hystrix Collapser, a custom BatchCollapser, and Guava's ConcurrentHashMultiset can dramatically reduce downstream load, improve system throughput, and outlines their implementations, configurations, and ideal use‑cases.

Batch ProcessingHystrixJava concurrency
0 likes · 15 min read
Merge Duplicate Requests Using Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset
Baidu Geek Talk
Baidu Geek Talk
Sep 24, 2025 · Big Data

How Feed Real‑Time Data Warehouse Was Re‑Engineered for Speed and Cost Savings

This article explains how Baidu’s Feed real‑time data warehouse was rebuilt using a pure streaming architecture, detailing the limitations of the previous stream‑batch design, the technical solutions—including core/non‑core data separation, metric calculation in streaming, and Parquet storage with Apache Arrow—and the resulting cost reductions, latency improvements, and future roadmap.

Apache ArrowBatch ProcessingParquet
0 likes · 17 min read
How Feed Real‑Time Data Warehouse Was Re‑Engineered for Speed and Cost Savings
Architect's Guide
Architect's Guide
Sep 18, 2025 · Backend Development

Master Spring Batch: From Basics to Advanced Job Configurations

This article provides a comprehensive guide to Spring Batch, covering its purpose, supported business scenarios, core components and interfaces, Maven setup, sample job definitions, parallel execution, decision flows, nested jobs, data reading and writing, validation processing, and REST‑based job scheduling, all illustrated with complete code examples.

Batch ProcessingJavaJob Scheduling
0 likes · 15 min read
Master Spring Batch: From Basics to Advanced Job Configurations
Architect's Guide
Architect's Guide
Aug 22, 2025 · Backend Development

Can JDBC Batch Insertion Speed Up MySQL? 100M Row Performance Test

This article evaluates the efficiency of inserting massive data into MySQL using Java, comparing three strategies—MyBatis without transactions, plain JDBC with and without transactions, and JDBC batch processing—showing that batch processing combined with transactions yields the fastest insertion speed for hundreds of millions of rows.

Batch ProcessingJDBCJava
0 likes · 13 min read
Can JDBC Batch Insertion Speed Up MySQL? 100M Row Performance Test
mikechen
mikechen
Aug 6, 2025 · Big Data

How to Diagnose and Fix Kafka Message Backlog in High‑Concurrency Environments

In high‑concurrency systems, Kafka message backlog occurs when producers outpace consumers, leading to unprocessed messages that threaten stability and real‑time performance, and this article explains the root causes and provides practical producer‑side and consumer‑side optimization techniques to resolve the issue.

Batch Processingconsumer optimizationhigh concurrency
0 likes · 5 min read
How to Diagnose and Fix Kafka Message Backlog in High‑Concurrency Environments
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Jul 22, 2025 · Big Data

How Apache SeaTunnel Revolutionizes Heterogeneous Data Integration with Decoupled Connectors

This article explores how Apache SeaTunnel addresses modern data integration challenges by providing a high‑performance, distributed, plugin‑based platform that decouples connectors from execution engines, enabling seamless batch and streaming synchronization across heterogeneous sources such as databases, message queues, and data lakes.

Apache SeaTunnelBatch ProcessingConnector Architecture
0 likes · 24 min read
How Apache SeaTunnel Revolutionizes Heterogeneous Data Integration with Decoupled Connectors
macrozheng
macrozheng
Jul 14, 2025 · Backend Development

Master Spring Batch: Core Concepts, Architecture, and Best Practices

This article provides a comprehensive guide to Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, and practical tips for configuring and optimizing batch jobs in Java applications.

Batch ProcessingChunkJava
0 likes · 21 min read
Master Spring Batch: Core Concepts, Architecture, and Best Practices
Instant Consumer Technology Team
Instant Consumer Technology Team
Jul 9, 2025 · Cloud Native

Scaling a Financial Accounting System to 100k TPS with Cloud‑Native Microservices

This article examines how a ten‑year‑old financial accounting platform transformed from a monolithic design into a cloud‑native, micro‑service architecture that achieved massive scalability, high availability, and 24‑hour real‑time processing through distributed batch scheduling, elastic scaling, and intelligent fault‑tolerance.

Batch Processingcloud-nativefinancial systems
0 likes · 14 min read
Scaling a Financial Accounting System to 100k TPS with Cloud‑Native Microservices
Code Ape Tech Column
Code Ape Tech Column
Jul 8, 2025 · Backend Development

Mastering Spring Batch: Real-World Use Cases and Hands‑On Guide

This comprehensive guide explains why batch processing is essential, walks through typical banking, e‑commerce, logging and medical data scenarios, details Spring Batch's core architecture and components, provides step‑by‑step setup and code examples, and presents a production‑grade bank reconciliation case with monitoring and troubleshooting tips.

Batch ProcessingData IntegrationJob Scheduling
0 likes · 27 min read
Mastering Spring Batch: Real-World Use Cases and Hands‑On Guide
Top Architect
Top Architect
Jul 5, 2025 · Databases

How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s with Batch, RewriteBatchedStatements, and Multithreading

This article walks through optimizing a 60,000‑row XML‑to‑MySQL import by profiling the environment, measuring baseline performance, and applying JDBC batch rewriting, write aggregation, and asynchronous writes with LMAX Disruptor, ultimately reducing execution time from 300 seconds to about four seconds while keeping memory usage reasonable.

Batch ProcessingDisruptorJDBC
0 likes · 15 min read
How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s with Batch, RewriteBatchedStatements, and Multithreading
Java Interview Crash Guide
Java Interview Crash Guide
Jul 3, 2025 · Backend Development

10 Proven Strategies to Supercharge API Performance in Java

This article presents a comprehensive, step‑by‑step guide to optimizing API latency by applying batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, code refactoring, pagination, SQL tuning, and fine‑grained locking techniques.

API optimizationBatch ProcessingCaching
0 likes · 10 min read
10 Proven Strategies to Supercharge API Performance in Java
Code Ape Tech Column
Code Ape Tech Column
Jun 30, 2025 · Backend Development

Mastering Software Performance: 6 Time‑Space Trade‑offs and 4 Advanced Parallel Techniques

This article explores practical performance‑optimization techniques, covering six fundamental time‑for‑space trade‑offs such as indexing, compression, caching, prefetching, peak‑shaving, and batch processing, followed by four advanced methods that boost parallelism like resource draining, horizontal scaling, sharding, and lock‑free programming.

BackendBatch ProcessingCaching
0 likes · 22 min read
Mastering Software Performance: 6 Time‑Space Trade‑offs and 4 Advanced Parallel Techniques
Architect's Tech Stack
Architect's Tech Stack
Jun 22, 2025 · Backend Development

Boost API Performance: 12 Proven Backend Optimization Techniques

This article presents a comprehensive set of twelve backend optimization strategies—including batch processing, asynchronous execution, caching, pooling, parallelism, indexing, transaction management, and SQL tuning—to dramatically reduce API latency and improve overall system efficiency.

API optimizationBackend PerformanceBatch Processing
0 likes · 9 min read
Boost API Performance: 12 Proven Backend Optimization Techniques
Java Captain
Java Captain
Jun 10, 2025 · Backend Development

Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide

This article explains the necessity of batch processing, presents typical use cases such as daily interest calculation, e‑commerce order archiving, log analysis and medical data migration, then dives deep into Spring Batch's core components, provides step‑by‑step code examples, performance‑tuning tips, production‑grade fault‑tolerance, monitoring solutions and a comprehensive FAQ.

Batch ProcessingData IntegrationJava
0 likes · 20 min read
Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide
Senior Tony
Senior Tony
Jun 8, 2025 · Backend Development

Why Kafka Handles Millions of Messages Per Second: Batch, Partition, Zero‑Copy, and Compression Explained

This article breaks down the core techniques that give Kafka its high‑throughput capability, including producer batch settings (batch.size, linger.ms), broker append‑only writes, consumer poll configuration, partition distribution, zero‑copy data transfer, dual‑thread processing, and configurable compression algorithms.

Batch ProcessingCompressionConsumer
0 likes · 8 min read
Why Kafka Handles Millions of Messages Per Second: Batch, Partition, Zero‑Copy, and Compression Explained
Tencent Cloud Developer
Tencent Cloud Developer
May 8, 2025 · Big Data

How Setats Unifies Stream, Batch, and Incremental Processing for Real‑Time Data Lakes

At the 2025 DA Data+AI Conference in Shanghai, Tencent Cloud unveiled Setats—a unified stream‑batch‑incremental engine that cuts system costs, delivers second‑level data visibility and real‑time changelog generation, and demonstrates measurable performance gains in automotive IoT analytics while integrating tightly with the WeData platform.

Batch ProcessingBig Data ArchitectureData Lake
0 likes · 5 min read
How Setats Unifies Stream, Batch, and Incremental Processing for Real‑Time Data Lakes
Wukong Talks Architecture
Wukong Talks Architecture
Apr 27, 2025 · Backend Development

Evolution of a Batch Processing System: From Centralized to Configurable and Localized Architectures

This article examines the evolution of a merchant batch processing system, detailing its transition from a centralized, tightly‑coupled architecture to a configurable, SPI‑based design and finally to a localized task‑reporting model, while discussing code reuse, scheduling strategies, isolation techniques, and performance challenges.

Batch ProcessingConfigurationJava
0 likes · 18 min read
Evolution of a Batch Processing System: From Centralized to Configurable and Localized Architectures
Architect
Architect
Apr 9, 2025 · Backend Development

Merging Requests and Batch Querying in Spring Boot to Reduce Database Connections

This article explains how to merge concurrent user requests into a single batch SQL query using Java's LinkedBlockingQueue, ScheduledThreadPoolExecutor and CompletableFuture in a Spring Boot application, thereby saving database connections and improving performance under high concurrency.

Batch ProcessingSpring Bootdatabase optimization
0 likes · 13 min read
Merging Requests and Batch Querying in Spring Boot to Reduce Database Connections
DataFunTalk
DataFunTalk
Apr 9, 2025 · Big Data

Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies

The Kuaishou‑hosted Apache Hudi Asia technical salon gathered over 230 attendees and featured seven experts from Kuaishou, Meituan, TikTok, Huawei, JD and others, who shared best practices, architecture designs, and performance optimizations for large‑scale data lake applications across AI, BI, and real‑time workloads.

AIApache HudiBatch Processing
0 likes · 14 min read
Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies
Su San Talks Tech
Su San Talks Tech
Apr 2, 2025 · Backend Development

How to Import Millions of Excel Rows in Seconds: 4 Proven Performance Hacks

This article analyzes why traditional Excel import methods crash under massive loads and presents four practical optimization techniques—including streaming parsing, batch inserts, asynchronous processing, and parallel sharding—backed by code samples, configuration tips, and real‑world performance benchmarks for importing millions of rows efficiently.

BackendBatch ProcessingJava
0 likes · 10 min read
How to Import Millions of Excel Rows in Seconds: 4 Proven Performance Hacks
JavaEdge
JavaEdge
Mar 30, 2025 · Artificial Intelligence

How GenAI Can Transform E‑Commerce Product Review Analysis

This article examines the critical role of product reviews for buyers and sellers, outlines the limitations of traditional review processing, and proposes a GenAI‑powered solution—including platform and model choices, batch inference, and semantic search—to efficiently analyze large‑scale e‑commerce feedback.

Batch ProcessingGenAINLP
0 likes · 12 min read
How GenAI Can Transform E‑Commerce Product Review Analysis
macrozheng
macrozheng
Mar 28, 2025 · Backend Development

Boost API Performance: 12 Proven Backend Optimization Techniques

This article presents a comprehensive set of backend optimization strategies—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, program refactoring, pagination, SQL tuning, and fine‑grained locking—to dramatically reduce API latency and improve system efficiency.

API optimizationBackend PerformanceBatch Processing
0 likes · 10 min read
Boost API Performance: 12 Proven Backend Optimization Techniques
Java Captain
Java Captain
Mar 21, 2025 · Backend Development

Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections

This article explains how to merge multiple user‑detail requests into a single database query using a blocking queue, scheduled thread pool, and CompletableFuture in Spring Boot, providing code examples, a high‑concurrency test, and discussion of trade‑offs such as added latency and timeout handling.

Batch ProcessingCompletableFutureConcurrency
0 likes · 13 min read
Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections
Architect's Tech Stack
Architect's Tech Stack
Mar 17, 2025 · Backend Development

Comprehensive Interface Performance Optimization Strategies

This article presents a systematic guide to improving API response times by applying batch processing, asynchronous execution, caching, pooling, parallelism, indexing, transaction management, pagination, SQL tuning, and proper lock granularity, supplemented with practical Java code examples and diagrams.

Batch ProcessingCachingLock Granularity
0 likes · 10 min read
Comprehensive Interface Performance Optimization Strategies
Java Backend Technology
Java Backend Technology
Mar 8, 2025 · Backend Development

How to Merge Concurrent Requests in Spring Boot and Save Database Connections

This article explains how to batch multiple user‑info requests on the server side, merge them into a single SQL query using a blocking queue and ScheduledThreadPoolExecutor, and return the results individually, thereby reducing database connection usage and improving performance under high concurrency.

Batch ProcessingCompletableFutureJava concurrency
0 likes · 13 min read
How to Merge Concurrent Requests in Spring Boot and Save Database Connections
macrozheng
macrozheng
Mar 5, 2025 · Backend Development

How to Merge Concurrent Requests in Spring Boot to Save Database Connections

This article explains how to combine multiple simultaneous user requests on the server side using a queue, scheduled thread pool and CompletableFuture in Spring Boot, reducing database connections while handling high concurrency, and discusses implementation details, testing, and potential pitfalls.

Batch ProcessingConcurrencySpring Boot
0 likes · 15 min read
How to Merge Concurrent Requests in Spring Boot to Save Database Connections
Top Architect
Top Architect
Jan 16, 2025 · Backend Development

Optimizing XML‑to‑MySQL Data Import: Reducing Execution Time from 300 s to 4 s

This article describes how to dramatically speed up the import of 60,000+ XML records into MySQL by analyzing the original environment, measuring baseline performance, and applying a series of backend optimizations—including MySQL batch mode, rewriteBatchedStatements, multithreaded asynchronous writes with Disruptor, and XML parsing improvements—ultimately cutting the total runtime from five minutes to just a few seconds.

Batch ProcessingDisruptorJava
0 likes · 14 min read
Optimizing XML‑to‑MySQL Data Import: Reducing Execution Time from 300 s to 4 s
Architect
Architect
Jan 12, 2025 · Databases

How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s

This article walks through a real‑world case of importing over 60,000 XML rows into MySQL, profiling the initial 300‑second runtime, and applying a series of optimizations—including JDBC batch processing, rewriteBatchedStatements, multithreaded asynchronous writes with Disruptor, and MySQL tuning—to achieve a final import time of around 4 seconds while managing memory usage.

Batch ProcessingDisruptorJava
0 likes · 13 min read
How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s
DataFunSummit
DataFunSummit
Nov 23, 2024 · Big Data

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

This article presents Bilibili's end‑to‑end exploration of a streaming‑batch unified data pipeline built on Apache Iceberg, detailing the original and iterated architectures for massive user behavior transmission, online AI training, DB synchronization, and dimension‑join, along with performance gains, cost savings, and future plans.

Batch ProcessingData LakeFlink
0 likes · 20 min read
Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice
ITPUB
ITPUB
Nov 1, 2024 · Backend Development

Why Our Nginx Data Gateway OOM’d: Tracing Memory Spikes & Core Dumps

An Nginx‑based data collection gateway began crashing with OOM kills, prompting a detailed investigation that uncovered memory spikes caused by aggressive batch processing, oversized protobuf payloads, and insufficient memory‑pool management, leading to a custom core‑dump solution and several mitigation strategies.

Batch ProcessingMemory LeakOOM
0 likes · 16 min read
Why Our Nginx Data Gateway OOM’d: Tracing Memory Spikes & Core Dumps
dbaplus Community
dbaplus Community
Oct 15, 2024 · Databases

How to Safely Perform Full-Table Updates on Billion-Row MySQL Tables

Updating billions of rows in a MySQL table can overwhelm binlog replication and cause deep‑pagination inefficiencies, so this article explains the pitfalls of direct UPDATE, explores limit‑based and IN‑based approaches, and presents a production‑ready batch update strategy using NO_CACHE and forced primary‑key indexing.

Batch ProcessingBinlogFull Table Update
0 likes · 7 min read
How to Safely Perform Full-Table Updates on Billion-Row MySQL Tables
Architecture Digest
Architecture Digest
Oct 11, 2024 · Backend Development

Common Interface Performance Optimization Techniques

This article outlines a series of backend interface performance optimization strategies—including batch processing, asynchronous execution, caching, preprocessing, pooling, parallelization, indexing, transaction management, program structure refactoring, deep pagination, SQL tuning, and lock granularity—to help developers reduce latency and improve system efficiency.

Batch ProcessingCachingIndexing
0 likes · 9 min read
Common Interface Performance Optimization Techniques
Programmer1970
Programmer1970
Sep 25, 2024 · Backend Development

How to Use XXL‑JOB in SpringBoot for Flexible Sharding of Massive Data

This article explains how to configure XXL‑JOB with a sharding‑broadcast strategy in a SpringBoot application, use ID modulo hashing to partition large tables, implement the task with XxlJobHelper, write MyBatis sharding SQL, and verify correct parallel processing through detailed logs.

Batch ProcessingMyBatisSpringBoot
0 likes · 7 min read
How to Use XXL‑JOB in SpringBoot for Flexible Sharding of Massive Data
JD Retail Technology
JD Retail Technology
Sep 25, 2024 · Big Data

From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design

The article narrates the author’s 30‑year programming career and shares technical reflections on building business‑agnostic, configurable data platforms, covering batch, streaming, interactive computing, big‑data sharding, Spark, Flink, cloud migration, and the philosophy of software architecture.

Batch ProcessingCloud ComputingSystem Design
0 likes · 23 min read
From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design
Su San Talks Tech
Su San Talks Tech
Sep 18, 2024 · Backend Development

Unlocking Kafka’s Million‑Message‑Per‑Second Performance: Key Techniques Explained

This article explains how Kafka attains extremely high throughput by using batch sending, message compression, sequential disk I/O, PageCache, zero‑copy transfers, and memory‑mapped index files, detailing the underlying code paths and trade‑offs that enable millions of messages per second.

Batch ProcessingCompressionHigh Performance
0 likes · 12 min read
Unlocking Kafka’s Million‑Message‑Per‑Second Performance: Key Techniques Explained
Architecture and Beyond
Architecture and Beyond
Sep 7, 2024 · Backend Development

Six Proven Backend Techniques to Supercharge System Performance

This comprehensive guide walks backend architects through six core optimization methods—caching, batch processing, asynchronous handling, data compression, parallelization, and eliminating unnecessary requests—detailing their problem domains, implementation strategies, real‑world scenarios, benefits, and trade‑offs.

BackendBatch ProcessingCaching
0 likes · 48 min read
Six Proven Backend Techniques to Supercharge System Performance
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 20, 2024 · Big Data

Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios

This article shares a personal, experience‑driven overview of Apache Paimon, highlighting its design simplicity, key capabilities such as schema evolution, stream‑batch unified processing, primary‑key support, and closed‑loop data handling, while discussing when its features are appropriate for production environments.

Apache PaimonBatch ProcessingStreaming
0 likes · 5 min read
Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Aug 16, 2024 · Big Data

Understanding the Lambda Architecture for Big Data Processing

This article explains the Lambda architecture—a three‑layer model combining batch and real‑time processing for large‑scale data, outlines its components, advantages, disadvantages, common tools, and compares it with the Kappa alternative while providing practical insights for data engineers.

Batch ProcessingLambda Architecturebig data
0 likes · 5 min read
Understanding the Lambda Architecture for Big Data Processing
FunTester
FunTester
Aug 15, 2024 · Backend Development

9 Proven Techniques to Supercharge Service Performance

This article outlines nine practical methods—caching, parallelization, batch processing, data compression, lock‑free design, sharding, request avoidance, pooling, and asynchronous handling—demonstrating how each can be applied to backend services to dramatically reduce latency and improve throughput.

Batch ProcessingCachingCompression
0 likes · 25 min read
9 Proven Techniques to Supercharge Service Performance