Skip to content
Tagged articles

batch processing

273 articles · Page 2 of 3
Architect
Architect
Jan 12, 2025 · Databases

How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s

This article walks through a real‑world case of importing over 60,000 XML rows into MySQL, profiling the initial 300‑second runtime, and applying a series of optimizations—including JDBC batch processing, rewriteBatchedStatements, multithreaded asynchronous writes with Disruptor, and MySQL tuning—to achieve a final import time of around 4 seconds while managing memory usage.

DisruptorJavaMultithreading
0 likes · 13 min read
How I Cut XML‑to‑MySQL Import Time from 300 s to 4 s
DataFunSummit
DataFunSummit
Nov 23, 2024 · Big Data

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

This article presents Bilibili's end‑to‑end exploration of a streaming‑batch unified data pipeline built on Apache Iceberg, detailing the original and iterated architectures for massive user behavior transmission, online AI training, DB synchronization, and dimension‑join, along with performance gains, cost savings, and future plans.

FlinkIcebergOptimization
0 likes · 20 min read
Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice
ITPUB
ITPUB
Nov 1, 2024 · Backend Development

Why Our Nginx Data Gateway OOM’d: Tracing Memory Spikes & Core Dumps

An Nginx‑based data collection gateway began crashing with OOM kills, prompting a detailed investigation that uncovered memory spikes caused by aggressive batch processing, oversized protobuf payloads, and insufficient memory‑pool management, leading to a custom core‑dump solution and several mitigation strategies.

Backend DebuggingMemory LeakOOM
0 likes · 16 min read
Why Our Nginx Data Gateway OOM’d: Tracing Memory Spikes & Core Dumps
dbaplus Community
dbaplus Community
Oct 15, 2024 · Databases

How to Safely Perform Full-Table Updates on Billion-Row MySQL Tables

Updating billions of rows in a MySQL table can overwhelm binlog replication and cause deep‑pagination inefficiencies, so this article explains the pitfalls of direct UPDATE, explores limit‑based and IN‑based approaches, and presents a production‑ready batch update strategy using NO_CACHE and forced primary‑key indexing.

Full Table UpdateSQLbatch processing
0 likes · 7 min read
How to Safely Perform Full-Table Updates on Billion-Row MySQL Tables
Architecture Digest
Architecture Digest
Oct 11, 2024 · Backend Development

Common Interface Performance Optimization Techniques

This article outlines a series of backend interface performance optimization strategies—including batch processing, asynchronous execution, caching, preprocessing, pooling, parallelization, indexing, transaction management, program structure refactoring, deep pagination, SQL tuning, and lock granularity—to help developers reduce latency and improve system efficiency.

IndexingOptimizationasynchronous
0 likes · 9 min read
Common Interface Performance Optimization Techniques
Programmer1970
Programmer1970
Sep 25, 2024 · Backend Development

How to Use XXL‑JOB in SpringBoot for Flexible Sharding of Massive Data

This article explains how to configure XXL‑JOB with a sharding‑broadcast strategy in a SpringBoot application, use ID modulo hashing to partition large tables, implement the task with XxlJobHelper, write MyBatis sharding SQL, and verify correct parallel processing through detailed logs.

MyBatisSpringBootXXL-JOB
0 likes · 7 min read
How to Use XXL‑JOB in SpringBoot for Flexible Sharding of Massive Data
JD Retail Technology
JD Retail Technology
Sep 25, 2024 · Big Data

From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design

The article narrates the author’s 30‑year programming career and shares technical reflections on building business‑agnostic, configurable data platforms, covering batch, streaming, interactive computing, big‑data sharding, Spark, Flink, cloud migration, and the philosophy of software architecture.

Data Engineeringbatch processingcloud computing
0 likes · 23 min read
From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design
Architecture and Beyond
Architecture and Beyond
Sep 7, 2024 · Backend Development

Six Proven Backend Techniques to Supercharge System Performance

This comprehensive guide walks backend architects through six core optimization methods—caching, batch processing, asynchronous handling, data compression, parallelization, and eliminating unnecessary requests—detailing their problem domains, implementation strategies, real‑world scenarios, benefits, and trade‑offs.

BackendCompressionParallelism
0 likes · 48 min read
Six Proven Backend Techniques to Supercharge System Performance
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 20, 2024 · Big Data

Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios

This article shares a personal, experience‑driven overview of Apache Paimon, highlighting its design simplicity, key capabilities such as schema evolution, stream‑batch unified processing, primary‑key support, and closed‑loop data handling, while discussing when its features are appropriate for production environments.

Apache PaimonSchema Evolutionbatch processing
0 likes · 5 min read
Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Aug 16, 2024 · Big Data

Understanding the Lambda Architecture for Big Data Processing

This article explains the Lambda architecture—a three‑layer model combining batch and real‑time processing for large‑scale data, outlines its components, advantages, disadvantages, common tools, and compares it with the Kappa alternative while providing practical insights for data engineers.

Data EngineeringLambda architecturebatch processing
0 likes · 5 min read
Understanding the Lambda Architecture for Big Data Processing
FunTester
FunTester
Aug 15, 2024 · Backend Development

9 Proven Techniques to Supercharge Service Performance

This article outlines nine practical methods—caching, parallelization, batch processing, data compression, lock‑free design, sharding, request avoidance, pooling, and asynchronous handling—demonstrating how each can be applied to backend services to dramatically reduce latency and improve throughput.

CompressionParallelismPerformance Optimization
0 likes · 25 min read
9 Proven Techniques to Supercharge Service Performance
DataFunSummit
DataFunSummit
Jul 12, 2024 · Big Data

Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design

This article examines the current evolution of data lakes, detailing their overall architecture, batch and real‑time integration methods, Lakehouse core functionalities such as enhanced DML, schema evolution, ACID support, and open‑design principles that enable multi‑cloud deployment and seamless interaction with diverse compute engines.

Open Data Formatsbatch processingbig data architecture
0 likes · 12 min read
Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design
DataFunSummit
DataFunSummit
May 17, 2024 · Big Data

Comprehensive Hudi Real-Time Data Lake Ingestion Solutions

This article presents a complete guide to Hudi-based real-time data lake ingestion, covering overall data integration architecture, batch and streaming ingestion strategies, advanced table design, and practical recommendations for handling challenges such as deduplication, latency, partitioning, and performance optimization.

HudiReal-time Ingestionbatch processing
0 likes · 12 min read
Comprehensive Hudi Real-Time Data Lake Ingestion Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 30, 2024 · Big Data

Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends

The article reviews Apache Paimon's graduation to an Apache Top-Level Project, outlines the essential capabilities of modern lakehouse frameworks—including streaming and batch I/O, multi‑engine integration, and advanced features—and discusses the problems they solve and the promising direction of the lakehouse ecosystem.

Apache PaimonReal-time Data Lakebatch processing
0 likes · 5 min read
Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends
Architect's Tech Stack
Architect's Tech Stack
Apr 29, 2024 · Databases

Performance Evaluation of Inserting Billion-Scale Data into MySQL Using MyBatis, JDBC, and Batch Processing

This article presents a comprehensive performance test of inserting massive amounts of randomly generated person records into MySQL, comparing three strategies—MyBatis lightweight insertion, direct JDBC handling, and JDBC batch processing—both with and without transactions, and concludes that combining batch processing with transactions yields the fastest insertion speed for large‑scale data loads.

JDBCLarge Data InsertionMyBatis
0 likes · 13 min read
Performance Evaluation of Inserting Billion-Scale Data into MySQL Using MyBatis, JDBC, and Batch Processing
Architect
Architect
Apr 8, 2024 · Backend Development

Mastering Batch Processing: Boost API Performance and Cut Overhead

This guide explains why batch processing is essential for API tuning and provides step‑by‑step techniques—including bulk database operations, request merging, pagination, parallel execution, caching, and monitoring—backed by concrete Java code samples and SQL queries to help engineers dramatically improve throughput and latency.

API optimizationDatabase PerformanceJava
0 likes · 33 min read
Mastering Batch Processing: Boost API Performance and Cut Overhead
DataFunSummit
DataFunSummit
Apr 7, 2024 · Big Data

Li Auto’s Flink on Kubernetes Data Integration Practice

This article presents Li Auto’s end‑to‑end data integration journey, detailing the evolution of its data platform, the challenges of heterogeneous sources, and how a unified Flink‑on‑K8s solution with cloud‑native architecture, operator management, monitoring, and checkpointing addresses batch‑stream convergence and future scalability.

FlinkKubernetesbatch processing
0 likes · 12 min read
Li Auto’s Flink on Kubernetes Data Integration Practice
FunTester
FunTester
Mar 21, 2024 · Operations

How Dynamic Task‑Grabbing Cuts Distributed Batch Jobs from Hours to Minutes

This article presents a detailed case study of optimizing a distributed batch processing system by replacing static shard‑key concurrency with a dynamic task‑grabbing mechanism, dramatically reducing execution time from several hours to under fifteen minutes while maintaining stable resource usage.

Distributed SystemsPerformance OptimizationTask Scheduling
0 likes · 8 min read
How Dynamic Task‑Grabbing Cuts Distributed Batch Jobs from Hours to Minutes
JD Retail Technology
JD Retail Technology
Feb 29, 2024 · Databases

Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution

This article presents a real‑world case study of optimizing massive batch‑processing tasks in an ad‑platform by applying query‑level improvements, cursor‑based pagination, shard‑aware batch updates, JVM‑tuned garbage collection, and distributed load‑balancing, ultimately reducing CPU usage from 80% to under 2% and cutting query‑per‑minute volume from millions to a few thousand.

Cursor PaginationJavabatch processing
0 likes · 22 min read
Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution
Python Crawling & Data Mining
Python Crawling & Data Mining
Feb 27, 2024 · Fundamentals

Automate Multi‑Sheet Excel Scoring with Python & Pandas: Step‑by‑Step Guide

This article walks through using Python and pandas to batch‑process seven Excel evaluation sheets, skipping header rows, cleaning data, computing total and average scores per person, merging results, and outputting aggregated statistics, providing a practical automation solution for repetitive office tasks.

Excel AutomationPythonbatch processing
0 likes · 7 min read
Automate Multi‑Sheet Excel Scoring with Python & Pandas: Step‑by‑Step Guide
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Feb 24, 2024 · Backend Development

Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing

This article introduces Karta, a lightweight Go library that provides two modes—Pipeline for unknown‑size asynchronous tasks and Group for known‑size batch tasks—offering a concise API, configurable workers, and built‑in callbacks to simplify high‑performance concurrent processing in backend applications.

Librarybatch processinggolang
0 likes · 9 min read
Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing
Python Crawling & Data Mining
Python Crawling & Data Mining
Jan 10, 2024 · Fundamentals

Batch Insert Images into Excel Cells with Python and openpyxl

This article demonstrates how to use Python's openpyxl library to automatically insert named images into specific Excel cells in bulk, explaining the problem, providing a complete script, and offering practical tips for handling large files and sharing the solution with the community.

Excel AutomationPythonbatch processing
0 likes · 5 min read
Batch Insert Images into Excel Cells with Python and openpyxl
FunTester
FunTester
Jan 9, 2024 · Databases

Boost Java Database Performance with PreparedStatement Batch Inserts

The article explains Java's PreparedStatement interface, highlighting its performance, security, and batch processing benefits, and provides a complete example of inserting multiple user records efficiently, while also discussing practical considerations such as batch size, transaction handling, error management, and optimization tips.

Database PerformanceJDBCJava
0 likes · 6 min read
Boost Java Database Performance with PreparedStatement Batch Inserts
DataFunTalk
DataFunTalk
Dec 22, 2023 · Big Data

Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto

This article details Li Auto's end‑to‑end data integration practice using Flink on Kubernetes, covering the evolution of their integration platform, architectural design, cloud‑native deployment, operational challenges, and future roadmap, while highlighting unified batch‑stream processing and resource elasticity.

Cloud NativeFlinkKubernetes
0 likes · 12 min read
Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto
DataFunTalk
DataFunTalk
Dec 18, 2023 · Big Data

Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing

The article explains why unified data architecture is essential to avoid duplication and inefficiency, discusses differing performance trade‑offs among batch, streaming, and interactive analytics, introduces an incremental computation model that unifies these modes, and invites readers to a Dec 19, 2023 technical sharing event.

Lambda architecturebatch processingbig data
0 likes · 3 min read
Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing
dbaplus Community
dbaplus Community
Dec 14, 2023 · Big Data

How Flink Powers Unified Stream‑Batch Processing at Scale: Production Lessons

This article explains why Flink was chosen as a unified stream‑batch engine, details the migration from Lambda architecture, outlines the Flink Batch production workflow, and shares key optimizations such as Hive dialect support, CTAS, adaptive scheduling, speculative execution, and future roadmap for large‑scale data processing.

Adaptive SchedulerFlinkHive Dialect
0 likes · 31 min read
How Flink Powers Unified Stream‑Batch Processing at Scale: Production Lessons
vivo Internet Technology
vivo Internet Technology
Dec 13, 2023 · Big Data

Hudi Data Lake Implementation and Optimization Practice at vivo

Vivo’s big‑data team deployed Apache Hudi to create a lakehouse that unifies streaming and batch workloads, leverages COW and MOR storage modes, automates small‑file clustering and compaction, and applies extensive version, streaming, batch, and lifecycle optimizations, delivering minute‑level latency, hundred‑million‑records‑per‑minute ingestion, and query speeds up to 20 % faster than Hive.

Apache HudiStreaming Processingbatch processing
0 likes · 11 min read
Hudi Data Lake Implementation and Optimization Practice at vivo
Selected Java Interview Questions
Selected Java Interview Questions
Dec 12, 2023 · Backend Development

Spring Boot Integration with Spring Batch: A Complete Tutorial and Example

This article provides a step‑by‑step guide on integrating Spring Batch with Spring Boot, covering business scenarios, database setup, Maven dependencies, configuration of JobRepository, JobLauncher, Job, Step, ItemReader, ItemProcessor, ItemWriter, listeners, validators, execution via REST endpoints, troubleshooting with Druid and switching to HikariCP, and demonstrates processing CSV and database data in large batches.

JavaMyBatisSpring Batch
0 likes · 19 min read
Spring Boot Integration with Spring Batch: A Complete Tutorial and Example
DaTaobao Tech
DaTaobao Tech
Dec 11, 2023 · Big Data

Design and Implementation of an Online Batch Processing Framework for Large-Scale Promotion Systems

The paper presents a centralized online batch‑processing framework for large‑scale promotion systems, where applications integrate via an SDK, a task‑center schedules and dispatches sub‑tasks through RocketMQ to Dubbo‑enabled containers, employing MapReduce‑style splitting, Guava rate‑limiting, heartbeat health checks, and has successfully handled over 1.3 million tasks during Double‑11.

Distributed SchedulingDubboJava
0 likes · 9 min read
Design and Implementation of an Online Batch Processing Framework for Large-Scale Promotion Systems
Java Architect Essentials
Java Architect Essentials
Dec 5, 2023 · Backend Development

Comprehensive Interface Performance Optimization Strategies

This article presents a collection of practical backend interface optimization techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, code restructuring, pagination, SQL tuning, and lock granularity—to reduce latency and improve overall system efficiency.

API optimizationasynchronousbatch processing
0 likes · 10 min read
Comprehensive Interface Performance Optimization Strategies
Architect
Architect
Nov 3, 2023 · Backend Development

How We Cut Risk Engine Latency from 80 ms to 25 ms with Prefetch, Batching, Async, Compression and Bloom‑Filter Caching

Facing a traffic surge that pushed a risk‑control engine's response time beyond 250 ms, the team applied a series of systematic optimizations—feature prefetching, batch requests, asynchronous accumulator updates, multi‑level caching with Bloom filters, and log‑compression redesign—resulting in latency dropping to 25 ms, CPU and memory usage falling by up to 90%, and storage costs reduced by over a third.

Asynchronous ProcessingBloom filterCompression
0 likes · 22 min read
How We Cut Risk Engine Latency from 80 ms to 25 ms with Prefetch, Batching, Async, Compression and Bloom‑Filter Caching
Bilibili Tech
Bilibili Tech
Oct 25, 2023 · Backend Development

Performance Optimization Practices in Bilibili's Risk Control Engine

To overcome storage, compute, and I/O bottlenecks in Bilibili’s risk‑control engine, the team combined pre‑fetching with Redis caching, batch retrieval, asynchronous writes via Railgun, aggressive log compression, and a multi‑level cache plus Bloom filter, cutting latency to sub‑100 ms, reducing Redis QPS by over 90 % and storage by ~38 %, while supporting million‑level query throughput.

BackendCompressionasync
0 likes · 22 min read
Performance Optimization Practices in Bilibili's Risk Control Engine
Java Backend Technology
Java Backend Technology
Oct 9, 2023 · Backend Development

How to Merge Requests in Spring Boot to Reduce DB Load and Boost Performance

This article explains how to combine multiple user queries into a single database request using a queue, ScheduledThreadPoolExecutor, and CompletableFuture in Spring Boot, demonstrating code implementations, handling Java 8 CompletableFuture timeout limitations, and showing performance gains through request merging under high concurrency.

ConcurrencyJavaSpring Boot
0 likes · 15 min read
How to Merge Requests in Spring Boot to Reduce DB Load and Boost Performance
Top Architect
Top Architect
Oct 8, 2023 · Backend Development

Merging Backend Requests in SpringBoot to Reduce Database Connections

This article explains how to merge multiple backend requests in a SpringBoot application using a blocking queue, ScheduledThreadPoolExecutor, and CompletableFuture to batch database queries, reduce connection overhead, handle high concurrency, and includes full Java code examples and performance testing.

Java ConcurrencySpringBootbackend optimization
0 likes · 15 min read
Merging Backend Requests in SpringBoot to Reduce Database Connections
Architect
Architect
Oct 1, 2023 · Backend Development

Batch Request Merging in Spring Boot to Reduce Database Connections

This article demonstrates how to merge multiple user‑info requests on the server side using a blocking queue, ScheduledThreadPoolExecutor, and CompletableFuture in Spring Boot, thereby consolidating SQL queries into a single batch call to save database connection resources while handling high concurrency.

CompletableFutureJava ConcurrencyLinkedBlockingQueue
0 likes · 13 min read
Batch Request Merging in Spring Boot to Reduce Database Connections
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 18, 2023 · Big Data

Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse

The article explains the mainstream Lambda data‑warehouse architecture, its benefits and challenges, then introduces Hudi as a lake‑house solution that unifies real‑time and offline storage, describes the multi‑layer service design, and showcases three practical scenarios—stream processing, real‑time multidimensional analysis, and stream‑batch data reuse—demonstrating how the integrated architecture improves latency, cost, and operational complexity.

HudiLambda architecturebatch processing
0 likes · 13 min read
Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse
Code Ape Tech Column
Code Ape Tech Column
Sep 16, 2023 · Backend Development

Batch Request Merging in Java to Reduce Database Connections

This article explains how to merge multiple user‑detail requests on the server side using a blocking queue, scheduled thread pool and CompletableFuture in Spring Boot, thereby converting many individual SQL calls into a single batch query, saving database connections and improving high‑concurrency performance.

CompletableFutureConcurrencyJava
0 likes · 13 min read
Batch Request Merging in Java to Reduce Database Connections
ITPUB
ITPUB
Sep 13, 2023 · Backend Development

Why Is Kafka So Fast? 7 Core Techniques Behind Its High Throughput

This article explains how Kafka achieves million‑message‑per‑second throughput by leveraging zero‑copy I/O, an append‑only log, batch processing, compression, consumer pull optimization, unflushed memory buffers, and JVM garbage‑collection tuning, detailing each mechanism and its impact on performance.

CompressionGC optimizationKafka
0 likes · 14 min read
Why Is Kafka So Fast? 7 Core Techniques Behind Its High Throughput
Architect's Tech Stack
Architect's Tech Stack
Sep 11, 2023 · Databases

Performance Evaluation of Large-Scale Data Insertion into MySQL Using MyBatis, JDBC, and Batch Processing

This article presents a systematic performance test of inserting massive data into MySQL, comparing three strategies—MyBatis lightweight insertion, direct JDBC (with and without transactions), and JDBC batch processing—showing how transaction handling and batch execution dramatically affect insertion speed.

JDBCJavaMyBatis
0 likes · 15 min read
Performance Evaluation of Large-Scale Data Insertion into MySQL Using MyBatis, JDBC, and Batch Processing
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 21, 2023 · Big Data

Key Features and Benefits of Lakehouse Frameworks Hudi, Iceberg, and Paimon

This note outlines how Hudi, Iceberg, and Paimon provide unified batch‑stream storage, UPSERT support, time‑travel capabilities, and lower development costs, enabling a streaming‑warehouse architecture that offers near‑real‑time latency, consistent semantics, persisted intermediate results, and easier historical data repair.

HudiIcebergPaimon
0 likes · 5 min read
Key Features and Benefits of Lakehouse Frameworks Hudi, Iceberg, and Paimon
Architect
Architect
Jul 30, 2023 · Backend Development

Spring Boot Integration with Spring Batch: A Comprehensive Tutorial

This article provides a step‑by‑step guide on integrating Spring Batch with Spring Boot, covering configuration, job and step setup, ItemReader/Processor/Writer implementations, database interactions, handling CSV and database sources, and troubleshooting common issues such as connection‑pool compatibility.

BackendJavaSpring Batch
0 likes · 22 min read
Spring Boot Integration with Spring Batch: A Comprehensive Tutorial
Senior Tony
Senior Tony
Jul 29, 2023 · Fundamentals

Essential Coding Habits Every Engineer Should Master

The article outlines practical coding habits—thorough input validation, comprehensive logging, careful RPC handling, batch processing, cautious SQL execution, safe extensions, disciplined refactoring, minimal dependencies, data consistency, and avoiding over‑engineering—to help engineers write reliable, maintainable code.

Data ConsistencyLoggingRPC
0 likes · 9 min read
Essential Coding Habits Every Engineer Should Master
Architect
Architect
Jul 10, 2023 · Big Data

Understanding Lambda Architecture for Real‑Time Billion‑Scale Data Analysis

This article explains the Lambda Architecture—a three‑layer big‑data processing model combining batch and speed layers to deliver accurate, low‑latency analytics, and illustrates its use with Twitter hashtag tracking and a smart‑parking recommendation system.

Lambda architectureServing LayerSpeed Layer
0 likes · 10 min read
Understanding Lambda Architecture for Real‑Time Billion‑Scale Data Analysis
DevOps
DevOps
Jul 4, 2023 · Operations

Best Practices for Batch Processing in Microservice Architecture

This article examines batch processing in microservice environments, defining its role versus data warehouses, exploring optimal placement across architectural layers, discussing access methods, and providing criteria such as complexity, consistency, security, performance, reliability, and scalability to guide the selection of an appropriate batch processing architecture.

architecturebatch processingdata synchronization
0 likes · 11 min read
Best Practices for Batch Processing in Microservice Architecture
Efficient Ops
Efficient Ops
Jun 13, 2023 · Operations

How to Automate Batch Job Retries and Eliminate Midnight Outages

This article explores a real‑world scenario where a support manager faces nightly batch job interruptions, analyzes common database and environment failures, and presents a systematic redesign of the batch framework and executor to enable automatic retry, reducing manual intervention and improving operational reliability.

Springbatch processingexception handling
0 likes · 7 min read
How to Automate Batch Job Retries and Eliminate Midnight Outages
Architecture Breakthrough
Architecture Breakthrough
May 8, 2023 · Backend Development

Designing a Robust Batch Processing Module: Key Architecture Insights

This article outlines the essential architectural considerations for building a production‑ready batch processing module, covering design principles, task scheduling, parallelism, error handling, resource management, data‑layer concerns, deployment strategies, and monitoring practices.

Backend ArchitectureJavaSpring Batch
0 likes · 10 min read
Designing a Robust Batch Processing Module: Key Architecture Insights
DataFunSummit
DataFunSummit
Apr 28, 2023 · Big Data

Building a Unified Streaming‑Batch Storage Architecture at Xiaohongshu

This article presents Xiaohongshu's design and implementation of a unified streaming‑batch storage system that integrates Lambda architecture, Kafka, Flink, Iceberg, and modern OLAP engines to solve real‑time data warehouse pain points and enable consistent, exactly‑once analytics across streaming and batch workloads.

FlinkIcebergKafka
0 likes · 16 min read
Building a Unified Streaming‑Batch Storage Architecture at Xiaohongshu
Java Architect Essentials
Java Architect Essentials
Apr 28, 2023 · Backend Development

12 Proven Strategies to Supercharge API Performance in Java

This article outlines twelve practical techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction sizing, code refactoring, pagination, SQL tuning, and lock granularity—to dramatically reduce API latency and improve overall backend efficiency.

API performanceIndexingJava
0 likes · 9 min read
12 Proven Strategies to Supercharge API Performance in Java
IT Architects Alliance
IT Architects Alliance
Apr 23, 2023 · Backend Development

Common Interface Performance Optimization Strategies

This article outlines a comprehensive set of backend interface optimization techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction sizing, code restructuring, deep pagination, SQL tuning, and lock granularity—to reduce latency and improve overall system efficiency.

API optimizationIndexingSQL Tuning
0 likes · 8 min read
Common Interface Performance Optimization Strategies
Top Architect
Top Architect
Apr 21, 2023 · Backend Development

Common Interface Performance Optimization Strategies

This article presents a comprehensive set of practical techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction size reduction, code refactoring, and pagination optimization—to systematically reduce API latency and improve overall backend service efficiency.

batch processingcachingdatabase
0 likes · 12 min read
Common Interface Performance Optimization Strategies
ByteDance Cloud Native
ByteDance Cloud Native
Apr 20, 2023 · Cloud Native

How Dragonfly Accelerates Image Distribution and Scales Kubernetes Batch Processing

At KubeCon+CloudNativeCon 2023 in Amsterdam, Volcano Engine and ByteDance presented two technical sessions covering Dragonfly's P2P image distribution best practices and large‑scale Kubernetes batch processing strategies, offering deep insights and real‑world implementations for cloud‑native developers.

DragonflyImage Distributionbatch processing
0 likes · 4 min read
How Dragonfly Accelerates Image Distribution and Scales Kubernetes Batch Processing
ITPUB
ITPUB
Mar 24, 2023 · Big Data

What’s New in Apache Flink 1.17? Key Features, Performance Gains, and Streaming Warehouse Advances

Apache Flink 1.17 introduces a suite of batch and streaming enhancements—including a new Streaming Warehouse API, significant TPC‑DS performance boosts, adaptive batch scheduling, improved checkpointing, expanded SQL capabilities, Hive connector upgrades, and broader filesystem support—while also delivering upgrades to FRocksDB, Calcite, and the token framework to strengthen its position as a leading unified data‑processing engine.

Apache FlinkRocksDBSQL
0 likes · 23 min read
What’s New in Apache Flink 1.17? Key Features, Performance Gains, and Streaming Warehouse Advances
DataFunTalk
DataFunTalk
Mar 12, 2023 · Big Data

Apache Kyuubi 1.6.0 Feature Overview and Enhancements

The article provides a comprehensive walkthrough of Apache Kyuubi 1.6.0, detailing server‑side enhancements such as batch (JAR) task submission, metadata store and unified API/authentication, client‑side improvements to the built‑in JDBC driver and Beeline, as well as engine plugins for Spark, Flink, Trino and Hive, and concludes with the community’s roadmap and statistics.

Apache KyuubiFlinkSQL Gateway
0 likes · 12 min read
Apache Kyuubi 1.6.0 Feature Overview and Enhancements
DataFunTalk
DataFunTalk
Feb 21, 2023 · Databases

Building a Stream‑Batch Integrated Data Architecture with Apache Doris at SelectDB

This article details how SelectDB’s data technology architect designed and implemented a new stream‑batch unified data platform using Apache Doris, covering the shortcomings of the early CDH‑based architecture, the selection process, data modeling, ingestion pipelines, performance testing, operational optimizations, and future plans.

Apache Dorisbatch processingbig data
0 likes · 17 min read
Building a Stream‑Batch Integrated Data Architecture with Apache Doris at SelectDB
DataFunTalk
DataFunTalk
Feb 2, 2023 · Big Data

SeaTunnel: Design Goals, Current Status, Architecture, and Future Roadmap

This article provides a comprehensive overview of Apache SeaTunnel, covering its design objectives, current capabilities such as multi‑engine support and extensive connector ecosystem, detailed architecture including engine‑independent APIs and execution flows, and outlines the upcoming roadmap to expand connectors, launch a visual web UI, and introduce a dedicated SeaTunnel Engine.

ApacheConnectorSeaTunnel
0 likes · 12 min read
SeaTunnel: Design Goals, Current Status, Architecture, and Future Roadmap
DataFunSummit
DataFunSummit
Jan 8, 2023 · Big Data

Streaming‑Batch Integrated Real‑time Multi‑dimensional Analysis

This article presents a comprehensive overview of evolving big‑data architectures—from classic offline warehouses to Lambda and Kappa models—and details a streaming‑batch integrated solution that addresses latency, data freshness, and multi‑table join challenges to achieve minute‑level real‑time multi‑dimensional analytics.

Kappa architectureLambda architecturebatch processing
0 likes · 18 min read
Streaming‑Batch Integrated Real‑time Multi‑dimensional Analysis
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 3, 2023 · Big Data

Migrating Hive SQL Jobs to Flink Using the SQL Gateway

This article explains how to use Apache Flink 1.16's SQL Gateway to migrate Hive SQL tasks to Flink, covering the underlying Hive‑on‑Flink architecture, dialect compatibility, streaming and batch demos, configuration details, and practical tips for developers and platform engineers.

FlinkHiveSQL Gateway
0 likes · 19 min read
Migrating Hive SQL Jobs to Flink Using the SQL Gateway
Architects' Tech Alliance
Architects' Tech Alliance
Dec 25, 2022 · Operations

Mastering Time‑Space Trade‑offs: Indexing, Caching, Compression & More

This article consolidates three performance‑optimization posts, detailing six universal time‑space trade‑off techniques—indexing, caching, compression, prefetching, peak‑valley smoothing, and batch processing—plus four advanced parallelism strategies, illustrated with real‑world analogies and practical guidelines for developers.

CompressionIndexingPerformance Optimization
0 likes · 22 min read
Mastering Time‑Space Trade‑offs: Indexing, Caching, Compression & More
Data Thinking Notes
Data Thinking Notes
Dec 21, 2022 · Big Data

Why Your Spark Batch Job Fails: Memory Limits, Data Skew, and Practical Fixes

This article examines a recurring Spark batch task failure caused by OutOfMemory errors and data skew, details the investigation steps—including increasing executor memory, raising parallelism, and analyzing shuffle metrics—and proposes solutions such as data validation, filtering oversized keys, and memory adjustments.

Data SkewOutOfMemorySpark
0 likes · 4 min read
Why Your Spark Batch Job Fails: Memory Limits, Data Skew, and Practical Fixes
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 15, 2022 · Big Data

Migrating Hive SQL to Flink SQL: Motivation, Challenges, Practice, Demo, and Future Plans

This technical article presents a comprehensive overview of migrating Hive SQL to Flink SQL, covering the motivations behind the migration, key challenges such as compatibility, stability and performance, practical implementation steps, a detailed demo, future development directions, and a Q&A session addressing common concerns.

FlinkHiveSQL migration
0 likes · 13 min read
Migrating Hive SQL to Flink SQL: Motivation, Challenges, Practice, Demo, and Future Plans
Top Architect
Top Architect
Dec 5, 2022 · Backend Development

Integrating Spring Batch with Spring Boot: Full Tutorial and Example

This article provides a comprehensive step‑by‑step guide on integrating Spring Batch with Spring Boot, covering configuration, entity and mapper definitions, job, step, reader, processor, writer implementations, database setup, error handling, and execution via a REST controller, complete with code examples.

JavaMyBatisSpring Batch
0 likes · 22 min read
Integrating Spring Batch with Spring Boot: Full Tutorial and Example
Java High-Performance Architecture
Java High-Performance Architecture
Dec 4, 2022 · Backend Development

How to Efficiently Import and Export Millions of Records with EasyExcel and POI

This article explains the challenges of large‑scale Excel import/export in Java, compares POI workbook implementations, introduces EasyExcel for high‑performance handling of hundreds of thousands to millions of rows, and provides detailed code examples for batch querying, sheet management, and JDBC batch insertion with transaction control.

EasyExcelJDBCJava
0 likes · 23 min read
How to Efficiently Import and Export Millions of Records with EasyExcel and POI
Top Architect
Top Architect
Nov 22, 2022 · Big Data

Efficient Massive Excel Import/Export with POI and EasyExcel in Java

This article explains how to efficiently import and export massive datasets (up to millions of rows) between Excel and databases using Apache POI, SXSSF, and Alibaba's EasyExcel, comparing workbook types, outlining performance considerations, and providing Java code examples for batch processing, paging, and transaction management.

EasyExcelExcelJava
0 likes · 23 min read
Efficient Massive Excel Import/Export with POI and EasyExcel in Java
ITPUB
ITPUB
Nov 18, 2022 · Big Data

How Xiaomi Uses Iceberg for Real‑Time Streaming and Batch Data Lakes

This article introduces Iceberg’s table‑format fundamentals, details Xiaomi’s large‑scale deployment of Iceberg for CDC and log ingestion, explores their streaming‑batch integration experiments, outlines future roadmap items, and provides a comprehensive Q&A covering practical challenges and solutions.

FlinkIcebergSpark
0 likes · 23 min read
How Xiaomi Uses Iceberg for Real‑Time Streaming and Batch Data Lakes
Architecture Digest
Architecture Digest
Nov 7, 2022 · Backend Development

Comprehensive Guide to Software Performance Optimization: Indexing, Compression, Caching, Prefetching, Throttling, and Batch Processing

This article presents a thorough, multi‑part exploration of software performance optimization techniques—including indexing, compression, caching, prefetching, peak‑shaving, and batch processing—explaining their principles, trade‑offs, practical applications, and how they relate to hardware constraints and system design.

CompressionIndexingOptimization
0 likes · 36 min read
Comprehensive Guide to Software Performance Optimization: Indexing, Compression, Caching, Prefetching, Throttling, and Batch Processing
Top Architect
Top Architect
Oct 20, 2022 · Databases

Performance Evaluation of Inserting Billion‑Scale Data into MySQL Using Java: MyBatis vs JDBC vs Batch Processing

This article presents a detailed performance test of inserting massive data into MySQL with Java, comparing three strategies—MyBatis lightweight insertion without transactions, direct JDBC handling with and without transactions, and JDBC batch processing—showing timing results for millions of rows and offering practical recommendations for high‑throughput data loading.

JDBCJavaMyBatis
0 likes · 14 min read
Performance Evaluation of Inserting Billion‑Scale Data into MySQL Using Java: MyBatis vs JDBC vs Batch Processing
DeWu Technology
DeWu Technology
Oct 10, 2022 · Big Data

Offline and Real-Time User Profile Fusion Architecture

The architecture combines a nightly batch job that generates offline user profiles stored in HBase with a Flink‑based stream layer that lazily loads those profiles on app start and creates real‑time updates, then fuses both streams into a unified, timestamp‑ordered profile in Redis, forming a Lambda‑style pipeline.

FlinkHBaseRedis
0 likes · 10 min read
Offline and Real-Time User Profile Fusion Architecture
Architect's Tech Stack
Architect's Tech Stack
Oct 9, 2022 · Backend Development

Spring Batch Overview: Architecture, Core Concepts, and Practical Usage

This article provides a comprehensive introduction to Spring Batch, covering its purpose for large‑scale data processing, overall architecture, key concepts such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip policies, and practical configuration examples with Java code.

JavaSpring BatchSpring Framework
0 likes · 17 min read
Spring Batch Overview: Architecture, Core Concepts, and Practical Usage
dbaplus Community
dbaplus Community
Sep 6, 2022 · Backend Development

How to Scale High‑Concurrency Write Operations with Asynchronous Patterns

This article examines a high‑traffic video‑watching scenario, identifies database write bottlenecks, and presents four asynchronous solutions—thread‑pool, in‑memory queue with scheduled tasks, message‑queue, and Agent‑plus‑MQ—detailing their implementation steps, trade‑offs, and best‑practice guidelines.

asynchronousbackend optimizationbatch processing
0 likes · 8 min read
How to Scale High‑Concurrency Write Operations with Asynchronous Patterns
DataFunTalk
DataFunTalk
Aug 25, 2022 · Big Data

Applying OpenMLDB for Efficient AI Toolchain and Data‑Driven Architecture at Akulaku

This article presents Akulaku’s practical experience with OpenMLDB, describing the company’s data‑driven requirements, the design of a unified stream‑batch architecture, implementation details across offline, online and RocksDB modes, and future recommendations for high‑performance, scenario‑agnostic big‑data processing.

AIOpenMLDBbatch processing
0 likes · 17 min read
Applying OpenMLDB for Efficient AI Toolchain and Data‑Driven Architecture at Akulaku
DaTaobao Tech
DaTaobao Tech
Aug 11, 2022 · Big Data

Unify SQL Engine: Integrating Stream, Batch, and Online Computing for Data Warehousing

The article describes how fragmented real‑time, batch, and online data‑warehouse pipelines suffer from low productivity and inconsistent data quality, and introduces a unified SQL engine built on Apache Calcite that parses, optimizes, and compiles a single SQL statement into executable plans for ODPS, Flink, or Java, leveraging Janino code generation, multi‑backend state storage, and snapshot‑join semantics to boost performance and simplify development.

CalciteFlinkSQL Engine
0 likes · 16 min read
Unify SQL Engine: Integrating Stream, Batch, and Online Computing for Data Warehousing
Java High-Performance Architecture
Java High-Performance Architecture
Aug 1, 2022 · Backend Development

Master Spring Batch: From Basics to Advanced Architecture and Best Practices

This article introduces Spring Batch, a lightweight Java batch‑processing framework, explains its architecture and core concepts such as Job, Step, ItemReader/Writer/Processor, chunk handling, skip strategies, and provides practical guidelines for building reliable, high‑throughput batch jobs while avoiding common pitfalls like memory exhaustion.

ChunkJavaJob
0 likes · 18 min read
Master Spring Batch: From Basics to Advanced Architecture and Best Practices
Programmer DD
Programmer DD
Jul 22, 2022 · Backend Development

Master Spring Batch: From Core Concepts to Advanced Chunk Processing

This article provides a comprehensive introduction to Spring Batch, covering its purpose, core architecture, key concepts such as Job, Step, ItemReader/Processor/Writer, chunk processing, skip handling, best practices, and common pitfalls like memory exhaustion, all illustrated with code examples and diagrams.

ChunkSkip StrategySpring Batch
0 likes · 20 min read
Master Spring Batch: From Core Concepts to Advanced Chunk Processing
Code Ape Tech Column
Code Ape Tech Column
Jul 16, 2022 · Backend Development

Spring Batch Architecture Overview and Core Concepts

This article introduces Spring Batch as a lightweight, comprehensive batch‑processing framework for enterprise applications, explains its overall architecture, and details core concepts such as Job, JobInstance, JobParameters, JobExecution, Step, StepExecution, ExecutionContext, JobRepository, JobLauncher, ItemReader, ItemWriter, ItemProcessor, chunk processing, skip/failure handling, best‑practice guidelines, and common troubleshooting tips.

ChunkItemReaderJava
0 likes · 20 min read
Spring Batch Architecture Overview and Core Concepts
Java Backend Technology
Java Backend Technology
Jul 12, 2022 · Backend Development

Boost Java Batch Updates with Multithreading: A Practical Guide

This article explains how to efficiently handle large‑scale batch updates in Java by splitting the data set, using a thread pool for concurrent processing, and providing reusable utility code, all illustrated with a clear workflow diagram and step‑by‑step examples.

BackendJavaMultithreading
0 likes · 7 min read
Boost Java Batch Updates with Multithreading: A Practical Guide
DataFunTalk
DataFunTalk
May 23, 2022 · Big Data

Real-Time Data Lake Practices at ByteDance: Architecture, Challenges, and Solutions

ByteDance shares its real‑time data lake implementation, covering the evolving definition of data lakes, six core capabilities, challenges such as data management, weak concurrent updates, performance, and log ingestion, and detailed solutions including Hudi Metastore Server, bucket indexing, multi‑stage use cases, and future roadmap.

HudiReal-time Data Lakebatch processing
0 likes · 32 min read
Real-Time Data Lake Practices at ByteDance: Architecture, Challenges, and Solutions