BestHub
Sign in
Tagged articles
224 articles
Page 1 of 3
Architect's Guide
Architect's Guide
Apr 27, 2026 · Backend Development

How to Optimize Large Transactions in API Endpoints

The article examines why large database transactions inside API endpoints degrade performance, outlines issues such as data inconsistency, lock contention, undo‑log overhead, and DB pressure, and presents practical optimizations including removing remote RPC calls, using programmatic transactions, batch processing, splitting into smaller transactions, and asynchronous parallel execution with code examples.

Batch ProcessingProgrammatic TransactionTransactions
0 likes · 9 min read
How to Optimize Large Transactions in API Endpoints
JD Tech
JD Tech
Apr 16, 2026 · Industry Insights

How JD Revolutionized Coupon Search with a Stream‑Batch Unified Architecture

This article analyzes JD's end‑to‑end upgrade of its retail coupon search infrastructure, detailing the business drivers, data‑skew challenges, the shift from dual KV and batch pipelines to a unified stream‑batch model built on Apache Doris, and the resulting performance, resource and stability gains across multiple scenarios.

Apache DorisBatch ProcessingCoupon Search
0 likes · 12 min read
How JD Revolutionized Coupon Search with a Stream‑Batch Unified Architecture
AI Explorer
AI Explorer
Mar 16, 2026 · Artificial Intelligence

How to Unlock NotebookLM’s Python API for Batch Automation and AI Workflows

The unofficial notebooklm-py library provides a Python API and CLI that give programmatic access to Google NotebookLM’s hidden capabilities, enabling batch import of PDFs or videos, automated generation of podcasts, mind‑maps, and other outputs, and integration into AI agents for scalable research workflows.

AI agentsAI automationBatch Processing
0 likes · 6 min read
How to Unlock NotebookLM’s Python API for Batch Automation and AI Workflows
Architect's Guide
Architect's Guide
Jan 16, 2026 · Databases

How to Safely Update Billions of MySQL Rows Without Overloading Binlog

This article explains why a naïve full‑table UPDATE on massive MySQL tables can cripple replication, explores deep‑pagination and IN‑clause inefficiencies, and presents a batch‑processing strategy using NO_CACHE, FORCE INDEX, and rate‑controlled scripts to perform safe, high‑performance updates.

Batch ProcessingBinlogFull Table Update
0 likes · 8 min read
How to Safely Update Billions of MySQL Rows Without Overloading Binlog
SpringMeng
SpringMeng
Jan 8, 2026 · Backend Development

How to Cut API Latency from Seconds to Milliseconds: Elegant Optimization Techniques

This article walks through a series of practical backend techniques—batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction management, pagination, and lock granularity—to dramatically reduce API response times from several seconds to just a few milliseconds.

API optimizationAsynchronousBatch Processing
0 likes · 9 min read
How to Cut API Latency from Seconds to Milliseconds: Elegant Optimization Techniques
DeWu Technology
DeWu Technology
Dec 25, 2025 · Frontend Development

Build a High‑Performance H5 PAG Player: SDK, Image Editing, Batch Synthesis

This guide details how to implement a full‑stack H5 PAG player for the “Use Basketball to Know Me” activity, covering SDK loading, canvas‑based image manipulation (drag, scale, rotate), dynamic layer and text replacement, real‑time preview synchronization, snapshot export, batch synthesis, performance tuning, and fallback strategies.

Batch ProcessingCanvasPAG
0 likes · 30 min read
Build a High‑Performance H5 PAG Player: SDK, Image Editing, Batch Synthesis
Architect's Tech Stack
Architect's Tech Stack
Dec 24, 2025 · Backend Development

10 Proven Techniques to Supercharge API Performance in Java Applications

This article presents a comprehensive, step‑by‑step guide to optimizing Java‑based API interfaces, covering batch processing, asynchronous execution, caching, pre‑processing, pooling, transaction handling, pagination, SQL tuning, lock granularity, and code restructuring, with practical code examples and diagrams.

API optimizationAsynchronousBatch Processing
0 likes · 10 min read
10 Proven Techniques to Supercharge API Performance in Java Applications
FunTester
FunTester
Dec 23, 2025 · Backend Development

Mastering Delayed, Priority, and Retry Tasks with River – A Go Queue Deep Dive

This article explains how River, a Go job‑queue library, implements delayed execution, priority handling, exponential‑backoff retries, batch inserts, monitoring, and best‑practice patterns, and compares it with other queue solutions to help developers build reliable, high‑performance background processing pipelines.

Batch ProcessingGoRiver
0 likes · 14 min read
Mastering Delayed, Priority, and Retry Tasks with River – A Go Queue Deep Dive
Ray's Galactic Tech
Ray's Galactic Tech
Dec 12, 2025 · Cloud Native

Mastering Kubernetes Jobs and CronJobs: Complete Guide & Practical Examples

Learn how Kubernetes Jobs and CronJobs enable one‑off and scheduled batch processing, understand their core concepts, key differences, YAML specifications, typical use cases, advanced configurations, monitoring, logging, and cleanup strategies, and see real‑world examples with complete YAML snippets and command‑line tips.

Batch ProcessingCloud NativeCronJob
0 likes · 8 min read
Mastering Kubernetes Jobs and CronJobs: Complete Guide & Practical Examples
Chen Tian Universe
Chen Tian Universe
Nov 27, 2025 · Fundamentals

A Comprehensive Overview of Modern Bank Core Systems Architecture

This article provides a detailed walkthrough of bank core system architecture, covering its evolution, global overview, key modules such as customer information, account types, deposit and loan cores, payment and settlement, general ledger, card management, and batch processing, illustrated with diagrams and real‑world examples.

Account ManagementBatch ProcessingLedger
0 likes · 20 min read
A Comprehensive Overview of Modern Bank Core Systems Architecture
Tech Freedom Circle
Tech Freedom Circle
Nov 16, 2025 · Databases

How Redis Pipeline Can Boost Performance 3‑12× and Impress Interviewers

This article explains Redis Pipeline’s core principle of batching commands to reduce network round‑trips, presents benchmark data showing up to 17‑fold speedups, details real‑world use cases such as cache warm‑up, heartbeat reporting, and high‑traffic events, and provides best‑practice guidelines on batch sizing, error handling, cluster constraints, and comparisons with transactions and Lua scripts.

Batch ProcessingDistributed SystemsPipeline
0 likes · 36 min read
How Redis Pipeline Can Boost Performance 3‑12× and Impress Interviewers
dbaplus Community
dbaplus Community
Nov 12, 2025 · Databases

Mastering Data Sync: From Full Loads to Real‑Time CDC in E‑Commerce

This guide walks a new e‑commerce developer through the evolution of order data synchronization—from naïve full‑table loads, through incremental and batch strategies, cursor‑based pagination, performance tuning, and finally to real‑time CDC with message queues—highlighting pitfalls and practical solutions.

Batch ProcessingCDCMessage Queue
0 likes · 12 min read
Mastering Data Sync: From Full Loads to Real‑Time CDC in E‑Commerce
Architect-Kip
Architect-Kip
Nov 4, 2025 · Backend Development

How to Optimize Batch Jobs for Better System and Database Performance

This article explains why batch tasks significantly affect system and database performance, and outlines three key strategies—task simplification, resource and performance tuning, and proactive monitoring with fault recovery—to design and optimize batch jobs effectively in enterprise development.

Batch ProcessingResource Isolationdatabase
0 likes · 4 min read
How to Optimize Batch Jobs for Better System and Database Performance
Su San Talks Tech
Su San Talks Tech
Oct 29, 2025 · Databases

How to Safely Delete Millions of Rows from a Large Table: Proven Strategies & Code

This article explains why deleting tens of millions of rows is challenging, analyzes transaction, lock, resource, and business impacts, and presents five practical solutions—including batch deletion, table rebuild, partition dropping, online migration, and professional tools—along with best‑practice guidelines and a decision matrix.

Batch ProcessingDatabase OptimizationPartitioning
0 likes · 28 min read
How to Safely Delete Millions of Rows from a Large Table: Proven Strategies & Code
Architect-Kip
Architect-Kip
Oct 28, 2025 · Operations

Mastering Failure Recovery: Fast‑Fail, Auto‑Retry, and Resilience Patterns for Distributed Systems

This guide outlines core principles and practical solutions for building resilient backend systems, covering fast‑failure handling, automatic retries with exponential back‑off, circuit‑breaker usage, idempotency, batch job strategies, online transaction patterns, and robust message‑queue processing.

Batch ProcessingIdempotencyMessage Queue
0 likes · 17 min read
Mastering Failure Recovery: Fast‑Fail, Auto‑Retry, and Resilience Patterns for Distributed Systems
Top Architect
Top Architect
Oct 28, 2025 · Backend Development

How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s

This article details a step‑by‑step performance overhaul for importing 60,000+ XML records into MySQL, covering baseline measurements, MySQL batch processing, asynchronous writes with Disruptor, XML parsing optimizations, and tuning MySQL buffers, ultimately reducing total runtime from 300 seconds to just four seconds.

Batch ProcessingDisruptorXML parsing
0 likes · 14 min read
How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s
Chen Tian Universe
Chen Tian Universe
Oct 14, 2025 · Operations

How Banks Turn Every Transaction into Precise Accounting Records

This article explains the core banking accounting system, covering the transaction‑driven entry engine, multidimensional accounting, internal account management, end‑of‑day batch processing, and a complete loan lifecycle example that shows how each step is recorded, reconciled, and reported.

BankingBatch Processingcore accounting
0 likes · 16 min read
How Banks Turn Every Transaction into Precise Accounting Records
Architect's Guide
Architect's Guide
Oct 3, 2025 · Backend Development

Merge Duplicate Requests Using Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset

This article explains how merging similar or duplicate requests upstream with Hystrix Collapser, a custom BatchCollapser, and Guava's ConcurrentHashMultiset can dramatically reduce downstream load, improve system throughput, and outlines their implementations, configurations, and ideal use‑cases.

Batch ProcessingHystrixJava concurrency
0 likes · 15 min read
Merge Duplicate Requests Using Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset
Baidu Geek Talk
Baidu Geek Talk
Sep 24, 2025 · Big Data

How Feed Real‑Time Data Warehouse Was Re‑Engineered for Speed and Cost Savings

This article explains how Baidu’s Feed real‑time data warehouse was rebuilt using a pure streaming architecture, detailing the limitations of the previous stream‑batch design, the technical solutions—including core/non‑core data separation, metric calculation in streaming, and Parquet storage with Apache Arrow—and the resulting cost reductions, latency improvements, and future roadmap.

Apache ArrowBatch ProcessingParquet
0 likes · 17 min read
How Feed Real‑Time Data Warehouse Was Re‑Engineered for Speed and Cost Savings
Architect's Guide
Architect's Guide
Sep 18, 2025 · Backend Development

Master Spring Batch: From Basics to Advanced Job Configurations

This article provides a comprehensive guide to Spring Batch, covering its purpose, supported business scenarios, core components and interfaces, Maven setup, sample job definitions, parallel execution, decision flows, nested jobs, data reading and writing, validation processing, and REST‑based job scheduling, all illustrated with complete code examples.

Batch ProcessingJob SchedulingSpring Batch
0 likes · 15 min read
Master Spring Batch: From Basics to Advanced Job Configurations
Architect's Guide
Architect's Guide
Aug 22, 2025 · Backend Development

Can JDBC Batch Insertion Speed Up MySQL? 100M Row Performance Test

This article evaluates the efficiency of inserting massive data into MySQL using Java, comparing three strategies—MyBatis without transactions, plain JDBC with and without transactions, and JDBC batch processing—showing that batch processing combined with transactions yields the fastest insertion speed for hundreds of millions of rows.

Batch ProcessingJDBCjava
0 likes · 13 min read
Can JDBC Batch Insertion Speed Up MySQL? 100M Row Performance Test
mikechen
mikechen
Aug 6, 2025 · Big Data

How to Diagnose and Fix Kafka Message Backlog in High‑Concurrency Environments

In high‑concurrency systems, Kafka message backlog occurs when producers outpace consumers, leading to unprocessed messages that threaten stability and real‑time performance, and this article explains the root causes and provides practical producer‑side and consumer‑side optimization techniques to resolve the issue.

Batch Processingconsumer optimizationhigh concurrency
0 likes · 5 min read
How to Diagnose and Fix Kafka Message Backlog in High‑Concurrency Environments
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Jul 22, 2025 · Big Data

How Apache SeaTunnel Revolutionizes Heterogeneous Data Integration with Decoupled Connectors

This article explores how Apache SeaTunnel addresses modern data integration challenges by providing a high‑performance, distributed, plugin‑based platform that decouples connectors from execution engines, enabling seamless batch and streaming synchronization across heterogeneous sources such as databases, message queues, and data lakes.

Apache SeaTunnelBatch ProcessingConnector Architecture
0 likes · 24 min read
How Apache SeaTunnel Revolutionizes Heterogeneous Data Integration with Decoupled Connectors
macrozheng
macrozheng
Jul 14, 2025 · Backend Development

Master Spring Batch: Core Concepts, Architecture, and Best Practices

This article provides a comprehensive guide to Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, and practical tips for configuring and optimizing batch jobs in Java applications.

Batch ProcessingChunkJob
0 likes · 21 min read
Master Spring Batch: Core Concepts, Architecture, and Best Practices
Instant Consumer Technology Team
Instant Consumer Technology Team
Jul 9, 2025 · Cloud Native

Scaling a Financial Accounting System to 100k TPS with Cloud‑Native Microservices

This article examines how a ten‑year‑old financial accounting platform transformed from a monolithic design into a cloud‑native, micro‑service architecture that achieved massive scalability, high availability, and 24‑hour real‑time processing through distributed batch scheduling, elastic scaling, and intelligent fault‑tolerance.

Batch ProcessingScalabilitycloud-native
0 likes · 14 min read
Scaling a Financial Accounting System to 100k TPS with Cloud‑Native Microservices
Code Ape Tech Column
Code Ape Tech Column
Jul 8, 2025 · Backend Development

Mastering Spring Batch: Real-World Use Cases and Hands‑On Guide

This comprehensive guide explains why batch processing is essential, walks through typical banking, e‑commerce, logging and medical data scenarios, details Spring Batch's core architecture and components, provides step‑by‑step setup and code examples, and presents a production‑grade bank reconciliation case with monitoring and troubleshooting tips.

Batch ProcessingData IntegrationJob Scheduling
0 likes · 27 min read
Mastering Spring Batch: Real-World Use Cases and Hands‑On Guide
Top Architect
Top Architect
Jul 5, 2025 · Databases

How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s with Batch, RewriteBatchedStatements, and Multithreading

This article walks through optimizing a 60,000‑row XML‑to‑MySQL import by profiling the environment, measuring baseline performance, and applying JDBC batch rewriting, write aggregation, and asynchronous writes with LMAX Disruptor, ultimately reducing execution time from 300 seconds to about four seconds while keeping memory usage reasonable.

Batch ProcessingDisruptorJDBC
0 likes · 15 min read
How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s with Batch, RewriteBatchedStatements, and Multithreading
Java Interview Crash Guide
Java Interview Crash Guide
Jul 3, 2025 · Backend Development

10 Proven Strategies to Supercharge API Performance in Java

This article presents a comprehensive, step‑by‑step guide to optimizing API latency by applying batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, code refactoring, pagination, SQL tuning, and fine‑grained locking techniques.

API optimizationBatch ProcessingSQL Tuning
0 likes · 10 min read
10 Proven Strategies to Supercharge API Performance in Java
Su San Talks Tech
Su San Talks Tech
Jul 1, 2025 · Big Data

How to Build Lightweight Batch Jobs with Spring Batch: A Practical Guide

This article explains the need for lightweight batch processing, outlines a layered architecture and robustness strategies, and demonstrates how Spring Batch implements these concepts with clear interfaces, job management, and support for ignore, retry, and restart mechanisms.

Batch ProcessingRobustnessSpring Batch
0 likes · 10 min read
How to Build Lightweight Batch Jobs with Spring Batch: A Practical Guide
Code Ape Tech Column
Code Ape Tech Column
Jun 30, 2025 · Backend Development

Mastering Software Performance: 6 Time‑Space Trade‑offs and 4 Advanced Parallel Techniques

This article explores practical performance‑optimization techniques, covering six fundamental time‑for‑space trade‑offs such as indexing, compression, caching, prefetching, peak‑shaving, and batch processing, followed by four advanced methods that boost parallelism like resource draining, horizontal scaling, sharding, and lock‑free programming.

BackendBatch Processingcaching
0 likes · 22 min read
Mastering Software Performance: 6 Time‑Space Trade‑offs and 4 Advanced Parallel Techniques
Architect's Tech Stack
Architect's Tech Stack
Jun 22, 2025 · Backend Development

Boost API Performance: 12 Proven Backend Optimization Techniques

This article presents a comprehensive set of twelve backend optimization strategies—including batch processing, asynchronous execution, caching, pooling, parallelism, indexing, transaction management, and SQL tuning—to dramatically reduce API latency and improve overall system efficiency.

API optimizationAsynchronousBackend Performance
0 likes · 9 min read
Boost API Performance: 12 Proven Backend Optimization Techniques
Java Captain
Java Captain
Jun 10, 2025 · Backend Development

Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide

This article explains the necessity of batch processing, presents typical use cases such as daily interest calculation, e‑commerce order archiving, log analysis and medical data migration, then dives deep into Spring Batch's core components, provides step‑by‑step code examples, performance‑tuning tips, production‑grade fault‑tolerance, monitoring solutions and a comprehensive FAQ.

Batch ProcessingData IntegrationSpring Batch
0 likes · 20 min read
Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide
Senior Tony
Senior Tony
Jun 8, 2025 · Backend Development

Why Kafka Handles Millions of Messages Per Second: Batch, Partition, Zero‑Copy, and Compression Explained

This article breaks down the core techniques that give Kafka its high‑throughput capability, including producer batch settings (batch.size, linger.ms), broker append‑only writes, consumer poll configuration, partition distribution, zero‑copy data transfer, dual‑thread processing, and configurable compression algorithms.

Batch ProcessingConsumerHigh Throughput
0 likes · 8 min read
Why Kafka Handles Millions of Messages Per Second: Batch, Partition, Zero‑Copy, and Compression Explained
Tencent Cloud Developer
Tencent Cloud Developer
May 8, 2025 · Big Data

How Setats Unifies Stream, Batch, and Incremental Processing for Real‑Time Data Lakes

At the 2025 DA Data+AI Conference in Shanghai, Tencent Cloud unveiled Setats—a unified stream‑batch‑incremental engine that cuts system costs, delivers second‑level data visibility and real‑time changelog generation, and demonstrates measurable performance gains in automotive IoT analytics while integrating tightly with the WeData platform.

Batch ProcessingBig Data ArchitectureData Lake
0 likes · 5 min read
How Setats Unifies Stream, Batch, and Incremental Processing for Real‑Time Data Lakes
Wukong Talks Architecture
Wukong Talks Architecture
Apr 27, 2025 · Backend Development

Evolution of a Batch Processing System: From Centralized to Configurable and Localized Architectures

This article examines the evolution of a merchant batch processing system, detailing its transition from a centralized, tightly‑coupled architecture to a configurable, SPI‑based design and finally to a localized task‑reporting model, while discussing code reuse, scheduling strategies, isolation techniques, and performance challenges.

Batch ProcessingConfigurationSystem Design
0 likes · 18 min read
Evolution of a Batch Processing System: From Centralized to Configurable and Localized Architectures
Architect
Architect
Apr 9, 2025 · Backend Development

Merging Requests and Batch Querying in Spring Boot to Reduce Database Connections

This article explains how to merge concurrent user requests into a single batch SQL query using Java's LinkedBlockingQueue, ScheduledThreadPoolExecutor and CompletableFuture in a Spring Boot application, thereby saving database connections and improving performance under high concurrency.

Batch ProcessingDatabase OptimizationSpring Boot
0 likes · 13 min read
Merging Requests and Batch Querying in Spring Boot to Reduce Database Connections
DataFunTalk
DataFunTalk
Apr 9, 2025 · Big Data

Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies

The Kuaishou‑hosted Apache Hudi Asia technical salon gathered over 230 attendees and featured seven experts from Kuaishou, Meituan, TikTok, Huawei, JD and others, who shared best practices, architecture designs, and performance optimizations for large‑scale data lake applications across AI, BI, and real‑time workloads.

Apache HudiBatch ProcessingBig Data
0 likes · 14 min read
Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies
Su San Talks Tech
Su San Talks Tech
Apr 2, 2025 · Backend Development

How to Import Millions of Excel Rows in Seconds: 4 Proven Performance Hacks

This article analyzes why traditional Excel import methods crash under massive loads and presents four practical optimization techniques—including streaming parsing, batch inserts, asynchronous processing, and parallel sharding—backed by code samples, configuration tips, and real‑world performance benchmarks for importing millions of rows efficiently.

BackendBatch ProcessingExcel Import
0 likes · 10 min read
How to Import Millions of Excel Rows in Seconds: 4 Proven Performance Hacks
JavaEdge
JavaEdge
Mar 30, 2025 · Artificial Intelligence

How GenAI Can Transform E‑Commerce Product Review Analysis

This article examines the critical role of product reviews for buyers and sellers, outlines the limitations of traditional review processing, and proposes a GenAI‑powered solution—including platform and model choices, batch inference, and semantic search—to efficiently analyze large‑scale e‑commerce feedback.

Batch ProcessingGenAINLP
0 likes · 12 min read
How GenAI Can Transform E‑Commerce Product Review Analysis
macrozheng
macrozheng
Mar 28, 2025 · Backend Development

Boost API Performance: 12 Proven Backend Optimization Techniques

This article presents a comprehensive set of backend optimization strategies—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, program refactoring, pagination, SQL tuning, and fine‑grained locking—to dramatically reduce API latency and improve system efficiency.

API optimizationAsynchronousBackend Performance
0 likes · 10 min read
Boost API Performance: 12 Proven Backend Optimization Techniques
Java Captain
Java Captain
Mar 21, 2025 · Backend Development

Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections

This article explains how to merge multiple user‑detail requests into a single database query using a blocking queue, scheduled thread pool, and CompletableFuture in Spring Boot, providing code examples, a high‑concurrency test, and discussion of trade‑offs such as added latency and timeout handling.

Batch ProcessingCompletableFutureQueue
0 likes · 13 min read
Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections
Architect's Tech Stack
Architect's Tech Stack
Mar 17, 2025 · Backend Development

Comprehensive Interface Performance Optimization Strategies

This article presents a systematic guide to improving API response times by applying batch processing, asynchronous execution, caching, pooling, parallelism, indexing, transaction management, pagination, SQL tuning, and proper lock granularity, supplemented with practical Java code examples and diagrams.

AsynchronousBatch ProcessingSQL pagination
0 likes · 10 min read
Comprehensive Interface Performance Optimization Strategies
Java Backend Technology
Java Backend Technology
Mar 8, 2025 · Backend Development

How to Merge Concurrent Requests in Spring Boot and Save Database Connections

This article explains how to batch multiple user‑info requests on the server side, merge them into a single SQL query using a blocking queue and ScheduledThreadPoolExecutor, and return the results individually, thereby reducing database connection usage and improving performance under high concurrency.

Batch ProcessingCompletableFutureJava concurrency
0 likes · 13 min read
How to Merge Concurrent Requests in Spring Boot and Save Database Connections
macrozheng
macrozheng
Mar 5, 2025 · Backend Development

How to Merge Concurrent Requests in Spring Boot to Save Database Connections

This article explains how to combine multiple simultaneous user requests on the server side using a queue, scheduled thread pool and CompletableFuture in Spring Boot, reducing database connections while handling high concurrency, and discusses implementation details, testing, and potential pitfalls.

Batch ProcessingSpring Bootconcurrency
0 likes · 15 min read
How to Merge Concurrent Requests in Spring Boot to Save Database Connections
Top Architect
Top Architect
Jan 16, 2025 · Backend Development

Optimizing XML‑to‑MySQL Data Import: Reducing Execution Time from 300 s to 4 s

This article describes how to dramatically speed up the import of 60,000+ XML records into MySQL by analyzing the original environment, measuring baseline performance, and applying a series of backend optimizations—including MySQL batch mode, rewriteBatchedStatements, multithreaded asynchronous writes with Disruptor, and XML parsing improvements—ultimately cutting the total runtime from five minutes to just a few seconds.

Batch ProcessingDisruptorXML parsing
0 likes · 14 min read
Optimizing XML‑to‑MySQL Data Import: Reducing Execution Time from 300 s to 4 s
Architect
Architect
Jan 12, 2025 · Databases

How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s

This article walks through a real‑world case of importing over 60,000 XML rows into MySQL, profiling the initial 300‑second runtime, and applying a series of optimizations—including JDBC batch processing, rewriteBatchedStatements, multithreaded asynchronous writes with Disruptor, and MySQL tuning—to achieve a final import time of around 4 seconds while managing memory usage.

Batch ProcessingDisruptorXML Import
0 likes · 13 min read
How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s
DataFunSummit
DataFunSummit
Nov 23, 2024 · Big Data

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

This article presents Bilibili's end‑to‑end exploration of a streaming‑batch unified data pipeline built on Apache Iceberg, detailing the original and iterated architectures for massive user behavior transmission, online AI training, DB synchronization, and dimension‑join, along with performance gains, cost savings, and future plans.

Batch ProcessingData LakeFlink
0 likes · 20 min read
Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice
ITPUB
ITPUB
Nov 1, 2024 · Backend Development

Why Our Nginx Data Gateway OOM’d: Tracing Memory Spikes & Core Dumps

An Nginx‑based data collection gateway began crashing with OOM kills, prompting a detailed investigation that uncovered memory spikes caused by aggressive batch processing, oversized protobuf payloads, and insufficient memory‑pool management, leading to a custom core‑dump solution and several mitigation strategies.

Batch ProcessingOOMProtobuf
0 likes · 16 min read
Why Our Nginx Data Gateway OOM’d: Tracing Memory Spikes & Core Dumps
dbaplus Community
dbaplus Community
Oct 15, 2024 · Databases

How to Safely Perform Full-Table Updates on Billion-Row MySQL Tables

Updating billions of rows in a MySQL table can overwhelm binlog replication and cause deep‑pagination inefficiencies, so this article explains the pitfalls of direct UPDATE, explores limit‑based and IN‑based approaches, and presents a production‑ready batch update strategy using NO_CACHE and forced primary‑key indexing.

Batch ProcessingBinlogFull Table Update
0 likes · 7 min read
How to Safely Perform Full-Table Updates on Billion-Row MySQL Tables
Architecture Digest
Architecture Digest
Oct 11, 2024 · Backend Development

Common Interface Performance Optimization Techniques

This article outlines a series of backend interface performance optimization strategies—including batch processing, asynchronous execution, caching, preprocessing, pooling, parallelization, indexing, transaction management, program structure refactoring, deep pagination, SQL tuning, and lock granularity—to help developers reduce latency and improve system efficiency.

AsynchronousBatch Processingcaching
0 likes · 9 min read
Common Interface Performance Optimization Techniques
JD Retail Technology
JD Retail Technology
Sep 25, 2024 · Big Data

From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design

The article narrates the author’s 30‑year programming career and shares technical reflections on building business‑agnostic, configurable data platforms, covering batch, streaming, interactive computing, big‑data sharding, Spark, Flink, cloud migration, and the philosophy of software architecture.

Batch ProcessingSoftware ArchitectureSystem Design
0 likes · 23 min read
From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design
Architecture and Beyond
Architecture and Beyond
Sep 7, 2024 · Backend Development

Six Proven Backend Techniques to Supercharge System Performance

This comprehensive guide walks backend architects through six core optimization methods—caching, batch processing, asynchronous handling, data compression, parallelization, and eliminating unnecessary requests—detailing their problem domains, implementation strategies, real‑world scenarios, benefits, and trade‑offs.

AsynchronousBackendBatch Processing
0 likes · 48 min read
Six Proven Backend Techniques to Supercharge System Performance
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 20, 2024 · Big Data

Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios

This article shares a personal, experience‑driven overview of Apache Paimon, highlighting its design simplicity, key capabilities such as schema evolution, stream‑batch unified processing, primary‑key support, and closed‑loop data handling, while discussing when its features are appropriate for production environments.

Apache PaimonBatch ProcessingBig Data
0 likes · 5 min read
Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Aug 16, 2024 · Big Data

Understanding the Lambda Architecture for Big Data Processing

This article explains the Lambda architecture—a three‑layer model combining batch and real‑time processing for large‑scale data, outlines its components, advantages, disadvantages, common tools, and compares it with the Kappa alternative while providing practical insights for data engineers.

Batch ProcessingBig DataLambda architecture
0 likes · 5 min read
Understanding the Lambda Architecture for Big Data Processing
FunTester
FunTester
Aug 15, 2024 · Backend Development

9 Proven Techniques to Supercharge Service Performance

This article outlines nine practical methods—caching, parallelization, batch processing, data compression, lock‑free design, sharding, request avoidance, pooling, and asynchronous handling—demonstrating how each can be applied to backend services to dramatically reduce latency and improve throughput.

AsynchronousBatch ProcessingParallelism
0 likes · 25 min read
9 Proven Techniques to Supercharge Service Performance
DataFunSummit
DataFunSummit
Jul 12, 2024 · Big Data

Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design

This article examines the current evolution of data lakes, detailing their overall architecture, batch and real‑time integration methods, Lakehouse core functionalities such as enhanced DML, schema evolution, ACID support, and open‑design principles that enable multi‑cloud deployment and seamless interaction with diverse compute engines.

Batch ProcessingBig Data ArchitectureData Lake
0 likes · 12 min read
Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design
DataFunSummit
DataFunSummit
May 17, 2024 · Big Data

Comprehensive Hudi Real-Time Data Lake Ingestion Solutions

This article presents a complete guide to Hudi-based real-time data lake ingestion, covering overall data integration architecture, batch and streaming ingestion strategies, advanced table design, and practical recommendations for handling challenges such as deduplication, latency, partitioning, and performance optimization.

Batch ProcessingBig DataData Lake
0 likes · 12 min read
Comprehensive Hudi Real-Time Data Lake Ingestion Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 30, 2024 · Big Data

Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends

The article reviews Apache Paimon's graduation to an Apache Top-Level Project, outlines the essential capabilities of modern lakehouse frameworks—including streaming and batch I/O, multi‑engine integration, and advanced features—and discusses the problems they solve and the promising direction of the lakehouse ecosystem.

Apache PaimonBatch ProcessingBig Data
0 likes · 5 min read
Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends
Architect's Tech Stack
Architect's Tech Stack
Apr 29, 2024 · Databases

Performance Evaluation of Inserting Billion-Scale Data into MySQL Using MyBatis, JDBC, and Batch Processing

This article presents a comprehensive performance test of inserting massive amounts of randomly generated person records into MySQL, comparing three strategies—MyBatis lightweight insertion, direct JDBC handling, and JDBC batch processing—both with and without transactions, and concludes that combining batch processing with transactions yields the fastest insertion speed for large‑scale data loads.

Batch ProcessingJDBCLarge Data Insertion
0 likes · 13 min read
Performance Evaluation of Inserting Billion-Scale Data into MySQL Using MyBatis, JDBC, and Batch Processing
Architect
Architect
Apr 8, 2024 · Backend Development

Mastering Batch Processing: Boost API Performance and Cut Overhead

This guide explains why batch processing is essential for API tuning and provides step‑by‑step techniques—including bulk database operations, request merging, pagination, parallel execution, caching, and monitoring—backed by concrete Java code samples and SQL queries to help engineers dramatically improve throughput and latency.

API optimizationBatch ProcessingDatabase Performance
0 likes · 33 min read
Mastering Batch Processing: Boost API Performance and Cut Overhead
DataFunSummit
DataFunSummit
Apr 7, 2024 · Big Data

Li Auto’s Flink on Kubernetes Data Integration Practice

This article presents Li Auto’s end‑to‑end data integration journey, detailing the evolution of its data platform, the challenges of heterogeneous sources, and how a unified Flink‑on‑K8s solution with cloud‑native architecture, operator management, monitoring, and checkpointing addresses batch‑stream convergence and future scalability.

Batch ProcessingBig DataData Integration
0 likes · 12 min read
Li Auto’s Flink on Kubernetes Data Integration Practice
FunTester
FunTester
Mar 21, 2024 · Operations

How Dynamic Task‑Grabbing Cuts Distributed Batch Jobs from Hours to Minutes

This article presents a detailed case study of optimizing a distributed batch processing system by replacing static shard‑key concurrency with a dynamic task‑grabbing mechanism, dramatically reducing execution time from several hours to under fifteen minutes while maintaining stable resource usage.

Batch ProcessingDistributed Systemsdynamic concurrency
0 likes · 8 min read
How Dynamic Task‑Grabbing Cuts Distributed Batch Jobs from Hours to Minutes
JD Retail Technology
JD Retail Technology
Feb 29, 2024 · Databases

Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution

This article presents a real‑world case study of optimizing massive batch‑processing tasks in an ad‑platform by applying query‑level improvements, cursor‑based pagination, shard‑aware batch updates, JVM‑tuned garbage collection, and distributed load‑balancing, ultimately reducing CPU usage from 80% to under 2% and cutting query‑per‑minute volume from millions to a few thousand.

Batch ProcessingDatabase Optimizationcursor pagination
0 likes · 22 min read
Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution
Python Crawling & Data Mining
Python Crawling & Data Mining
Feb 27, 2024 · Fundamentals

Automate Multi‑Sheet Excel Scoring with Python & Pandas: Step‑by‑Step Guide

This article walks through using Python and pandas to batch‑process seven Excel evaluation sheets, skipping header rows, cleaning data, computing total and average scores per person, merging results, and outputting aggregated statistics, providing a practical automation solution for repetitive office tasks.

Batch ProcessingExcel AutomationPython
0 likes · 7 min read
Automate Multi‑Sheet Excel Scoring with Python & Pandas: Step‑by‑Step Guide
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Feb 24, 2024 · Backend Development

Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing

This article introduces Karta, a lightweight Go library that provides two modes—Pipeline for unknown‑size asynchronous tasks and Group for known‑size batch tasks—offering a concise API, configurable workers, and built‑in callbacks to simplify high‑performance concurrent processing in backend applications.

Batch ProcessingGolangLibrary
0 likes · 9 min read
Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing
Python Crawling & Data Mining
Python Crawling & Data Mining
Jan 10, 2024 · Fundamentals

Batch Insert Images into Excel Cells with Python and openpyxl

This article demonstrates how to use Python's openpyxl library to automatically insert named images into specific Excel cells in bulk, explaining the problem, providing a complete script, and offering practical tips for handling large files and sharing the solution with the community.

Batch ProcessingExcel AutomationPython
0 likes · 5 min read
Batch Insert Images into Excel Cells with Python and openpyxl
FunTester
FunTester
Jan 9, 2024 · Databases

Boost Java Database Performance with PreparedStatement Batch Inserts

The article explains Java's PreparedStatement interface, highlighting its performance, security, and batch processing benefits, and provides a complete example of inserting multiple user records efficiently, while also discussing practical considerations such as batch size, transaction handling, error management, and optimization tips.

Batch ProcessingDatabase PerformanceJDBC
0 likes · 6 min read
Boost Java Database Performance with PreparedStatement Batch Inserts
DataFunTalk
DataFunTalk
Dec 22, 2023 · Big Data

Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto

This article details Li Auto's end‑to‑end data integration practice using Flink on Kubernetes, covering the evolution of their integration platform, architectural design, cloud‑native deployment, operational challenges, and future roadmap, while highlighting unified batch‑stream processing and resource elasticity.

Batch ProcessingBig DataCloud Native
0 likes · 12 min read
Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto
DataFunTalk
DataFunTalk
Dec 18, 2023 · Big Data

Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing

The article explains why unified data architecture is essential to avoid duplication and inefficiency, discusses differing performance trade‑offs among batch, streaming, and interactive analytics, introduces an incremental computation model that unifies these modes, and invites readers to a Dec 19, 2023 technical sharing event.

Batch ProcessingBig DataData Architecture
0 likes · 3 min read
Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing
dbaplus Community
dbaplus Community
Dec 14, 2023 · Big Data

How Flink Powers Unified Stream‑Batch Processing at Scale: Production Lessons

This article explains why Flink was chosen as a unified stream‑batch engine, details the migration from Lambda architecture, outlines the Flink Batch production workflow, and shares key optimizations such as Hive dialect support, CTAS, adaptive scheduling, speculative execution, and future roadmap for large‑scale data processing.

Adaptive SchedulerBatch ProcessingBig Data
0 likes · 31 min read
How Flink Powers Unified Stream‑Batch Processing at Scale: Production Lessons
vivo Internet Technology
vivo Internet Technology
Dec 13, 2023 · Big Data

Hudi Data Lake Implementation and Optimization Practice at vivo

Vivo’s big‑data team deployed Apache Hudi to create a lakehouse that unifies streaming and batch workloads, leverages COW and MOR storage modes, automates small‑file clustering and compaction, and applies extensive version, streaming, batch, and lifecycle optimizations, delivering minute‑level latency, hundred‑million‑records‑per‑minute ingestion, and query speeds up to 20 % faster than Hive.

Apache HudiBatch ProcessingBig Data
0 likes · 11 min read
Hudi Data Lake Implementation and Optimization Practice at vivo
Selected Java Interview Questions
Selected Java Interview Questions
Dec 12, 2023 · Backend Development

Spring Boot Integration with Spring Batch: A Complete Tutorial and Example

This article provides a step‑by‑step guide on integrating Spring Batch with Spring Boot, covering business scenarios, database setup, Maven dependencies, configuration of JobRepository, JobLauncher, Job, Step, ItemReader, ItemProcessor, ItemWriter, listeners, validators, execution via REST endpoints, troubleshooting with Druid and switching to HikariCP, and demonstrates processing CSV and database data in large batches.

Batch ProcessingMyBatisSpring Batch
0 likes · 19 min read
Spring Boot Integration with Spring Batch: A Complete Tutorial and Example
DaTaobao Tech
DaTaobao Tech
Dec 11, 2023 · Big Data

Design and Implementation of an Online Batch Processing Framework for Large-Scale Promotion Systems

The paper presents a centralized online batch‑processing framework for large‑scale promotion systems, where applications integrate via an SDK, a task‑center schedules and dispatches sub‑tasks through RocketMQ to Dubbo‑enabled containers, employing MapReduce‑style splitting, Guava rate‑limiting, heartbeat health checks, and has successfully handled over 1.3 million tasks during Double‑11.

Batch ProcessingBig DataDistributed Scheduling
0 likes · 9 min read
Design and Implementation of an Online Batch Processing Framework for Large-Scale Promotion Systems
Java Architect Essentials
Java Architect Essentials
Dec 5, 2023 · Backend Development

Comprehensive Interface Performance Optimization Strategies

This article presents a collection of practical backend interface optimization techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, code restructuring, pagination, SQL tuning, and lock granularity—to reduce latency and improve overall system efficiency.

API optimizationAsynchronousBatch Processing
0 likes · 10 min read
Comprehensive Interface Performance Optimization Strategies
Architect
Architect
Nov 3, 2023 · Backend Development

How We Cut Risk Engine Latency from 80 ms to 25 ms with Prefetch, Batching, Async, Compression and Bloom‑Filter Caching

Facing a traffic surge that pushed a risk‑control engine's response time beyond 250 ms, the team applied a series of systematic optimizations—feature prefetching, batch requests, asynchronous accumulator updates, multi‑level caching with Bloom filters, and log‑compression redesign—resulting in latency dropping to 25 ms, CPU and memory usage falling by up to 90%, and storage costs reduced by over a third.

Batch Processingasynchronous processingbackend systems
0 likes · 22 min read
How We Cut Risk Engine Latency from 80 ms to 25 ms with Prefetch, Batching, Async, Compression and Bloom‑Filter Caching
Architect's Guide
Architect's Guide
Nov 3, 2023 · Databases

Performance Evaluation of Inserting Hundreds of Millions of Records into MySQL Using Java, MyBatis, and JDBC

This article experimentally compares three Java‑based insertion strategies—MyBatis without transaction, plain JDBC with and without transaction, and JDBC batch processing—with and without transactions, measuring how they affect the time required to insert tens of millions of rows into MySQL.

Batch ProcessingDatabase OptimizationJDBC
0 likes · 13 min read
Performance Evaluation of Inserting Hundreds of Millions of Records into MySQL Using Java, MyBatis, and JDBC
Bilibili Tech
Bilibili Tech
Oct 25, 2023 · Backend Development

Performance Optimization Practices in Bilibili's Risk Control Engine

To overcome storage, compute, and I/O bottlenecks in Bilibili’s risk‑control engine, the team combined pre‑fetching with Redis caching, batch retrieval, asynchronous writes via Railgun, aggressive log compression, and a multi‑level cache plus Bloom filter, cutting latency to sub‑100 ms, reducing Redis QPS by over 90 % and storage by ~38 %, while supporting million‑level query throughput.

AsyncBackendBatch Processing
0 likes · 22 min read
Performance Optimization Practices in Bilibili's Risk Control Engine
Java Backend Technology
Java Backend Technology
Oct 9, 2023 · Backend Development

How to Merge Requests in Spring Boot to Reduce DB Load and Boost Performance

This article explains how to combine multiple user queries into a single database request using a queue, ScheduledThreadPoolExecutor, and CompletableFuture in Spring Boot, demonstrating code implementations, handling Java 8 CompletableFuture timeout limitations, and showing performance gains through request merging under high concurrency.

Batch ProcessingSpring Bootconcurrency
0 likes · 15 min read
How to Merge Requests in Spring Boot to Reduce DB Load and Boost Performance
Top Architect
Top Architect
Oct 8, 2023 · Backend Development

Merging Backend Requests in SpringBoot to Reduce Database Connections

This article explains how to merge multiple backend requests in a SpringBoot application using a blocking queue, ScheduledThreadPoolExecutor, and CompletableFuture to batch database queries, reduce connection overhead, handle high concurrency, and includes full Java code examples and performance testing.

Batch ProcessingJava concurrencySpringBoot
0 likes · 15 min read
Merging Backend Requests in SpringBoot to Reduce Database Connections
Architect
Architect
Oct 1, 2023 · Backend Development

Batch Request Merging in Spring Boot to Reduce Database Connections

This article demonstrates how to merge multiple user‑info requests on the server side using a blocking queue, ScheduledThreadPoolExecutor, and CompletableFuture in Spring Boot, thereby consolidating SQL queries into a single batch call to save database connection resources while handling high concurrency.

Batch ProcessingCompletableFutureJava concurrency
0 likes · 13 min read
Batch Request Merging in Spring Boot to Reduce Database Connections
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 18, 2023 · Big Data

Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse

The article explains the mainstream Lambda data‑warehouse architecture, its benefits and challenges, then introduces Hudi as a lake‑house solution that unifies real‑time and offline storage, describes the multi‑layer service design, and showcases three practical scenarios—stream processing, real‑time multidimensional analysis, and stream‑batch data reuse—demonstrating how the integrated architecture improves latency, cost, and operational complexity.

Batch ProcessingHudiLakehouse
0 likes · 13 min read
Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse
Code Ape Tech Column
Code Ape Tech Column
Sep 16, 2023 · Backend Development

Batch Request Merging in Java to Reduce Database Connections

This article explains how to merge multiple user‑detail requests on the server side using a blocking queue, scheduled thread pool and CompletableFuture in Spring Boot, thereby converting many individual SQL calls into a single batch query, saving database connections and improving high‑concurrency performance.

Batch ProcessingCompletableFutureDatabase Optimization
0 likes · 13 min read
Batch Request Merging in Java to Reduce Database Connections
ITPUB
ITPUB
Sep 13, 2023 · Backend Development

Why Is Kafka So Fast? 7 Core Techniques Behind Its High Throughput

This article explains how Kafka achieves million‑message‑per‑second throughput by leveraging zero‑copy I/O, an append‑only log, batch processing, compression, consumer pull optimization, unflushed memory buffers, and JVM garbage‑collection tuning, detailing each mechanism and its impact on performance.

Batch ProcessingGC optimizationKafka
0 likes · 14 min read
Why Is Kafka So Fast? 7 Core Techniques Behind Its High Throughput
Architect's Tech Stack
Architect's Tech Stack
Sep 11, 2023 · Databases

Performance Evaluation of Large-Scale Data Insertion into MySQL Using MyBatis, JDBC, and Batch Processing

This article presents a systematic performance test of inserting massive data into MySQL, comparing three strategies—MyBatis lightweight insertion, direct JDBC (with and without transactions), and JDBC batch processing—showing how transaction handling and batch execution dramatically affect insertion speed.

Batch ProcessingJDBCMyBatis
0 likes · 15 min read
Performance Evaluation of Large-Scale Data Insertion into MySQL Using MyBatis, JDBC, and Batch Processing