Batch Processing

Collection size

121 articles

Page 3 of 7

DataFunTalk

Apr 9, 2025 · Big Data

Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies

The Kuaishou‑hosted Apache Hudi Asia technical salon gathered over 230 attendees and featured seven experts from Kuaishou, Meituan, TikTok, Huawei, JD and others, who shared best practices, architecture designs, and performance optimizations for large‑scale data lake applications across AI, BI, and real‑time workloads.

AIApache HudiBatch Processing

0 likes · 14 min read

Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies

DataFunSummit

Nov 23, 2024 · Big Data

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

This article presents Bilibili's end‑to‑end exploration of a streaming‑batch unified data pipeline built on Apache Iceberg, detailing the original and iterated architectures for massive user behavior transmission, online AI training, DB synchronization, and dimension‑join, along with performance gains, cost savings, and future plans.

Batch ProcessingFlinkdata lake

0 likes · 20 min read

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

DataFunSummit

Jul 12, 2024 · Big Data

Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design

This article examines the current evolution of data lakes, detailing their overall architecture, batch and real‑time integration methods, Lakehouse core functionalities such as enhanced DML, schema evolution, ACID support, and open‑design principles that enable multi‑cloud deployment and seamless interaction with diverse compute engines.

Batch ProcessingOpen Data FormatsReal-time Integration

0 likes · 12 min read

Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design

DataFunSummit

Apr 7, 2024 · Big Data

Li Auto’s Flink on Kubernetes Data Integration Practice

This article presents Li Auto’s end‑to‑end data integration journey, detailing the evolution of its data platform, the challenges of heterogeneous sources, and how a unified Flink‑on‑K8s solution with cloud‑native architecture, operator management, monitoring, and checkpointing addresses batch‑stream convergence and future scalability.

Batch ProcessingCloud NativeFlink

0 likes · 12 min read

Li Auto’s Flink on Kubernetes Data Integration Practice

DataFunSummit

Apr 28, 2023 · Big Data

Building a Unified Streaming‑Batch Storage Architecture at Xiaohongshu

This article presents Xiaohongshu's design and implementation of a unified streaming‑batch storage system that integrates Lambda architecture, Kafka, Flink, Iceberg, and modern OLAP engines to solve real‑time data warehouse pain points and enable consistent, exactly‑once analytics across streaming and batch workloads.

Batch ProcessingFlinkKafka

0 likes · 16 min read

Building a Unified Streaming‑Batch Storage Architecture at Xiaohongshu

DataFunTalk

Dec 18, 2023 · Big Data

Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing

The article explains why unified data architecture is essential to avoid duplication and inefficiency, discusses differing performance trade‑offs among batch, streaming, and interactive analytics, introduces an incremental computation model that unifies these modes, and invites readers to a Dec 19, 2023 technical sharing event.

Batch ProcessingIncremental Computingbig data

0 likes · 3 min read

Unified Data Architecture: Balancing Freshness, Cost, and Performance with Incremental Computing

DataFunTalk

Mar 12, 2023 · Big Data

Apache Kyuubi 1.6.0 Feature Overview and Enhancements

The article provides a comprehensive walkthrough of Apache Kyuubi 1.6.0, detailing server‑side enhancements such as batch (JAR) task submission, metadata store and unified API/authentication, client‑side improvements to the built‑in JDBC driver and Beeline, as well as engine plugins for Spark, Flink, Trino and Hive, and concludes with the community’s roadmap and statistics.

Apache KyuubiBatch ProcessingFlink

0 likes · 12 min read

Apache Kyuubi 1.6.0 Feature Overview and Enhancements

DataFunTalk

May 23, 2022 · Big Data

Real-Time Data Lake Practices at ByteDance: Architecture, Challenges, and Solutions

ByteDance shares its real‑time data lake implementation, covering the evolving definition of data lakes, six core capabilities, challenges such as data management, weak concurrent updates, performance, and log ingestion, and detailed solutions including Hudi Metastore Server, bucket indexing, multi‑stage use cases, and future roadmap.

Batch ProcessingHudibig data

0 likes · 32 min read

Real-Time Data Lake Practices at ByteDance: Architecture, Challenges, and Solutions

Architect's Guide

May 25, 2023 · Backend Development

Spring Batch: Introduction, Architecture, Core Interfaces, and Practical Implementation Guide

This article provides a comprehensive overview of Spring Batch, covering its purpose, typical business scenarios, core components such as JobRepository, JobLauncher, Job, Step, and core interfaces like ItemReader, ItemProcessor, ItemWriter, followed by detailed code examples for configuration, multi‑step, parallel, decision, nested jobs, data reading, writing, processing, and scheduling using Spring Boot.

Backend DevelopmentBatch ProcessingJava

0 likes · 15 min read

Spring Batch: Introduction, Architecture, Core Interfaces, and Practical Implementation Guide

Selected Java Interview Questions

Dec 12, 2023 · Backend Development

Spring Boot Integration with Spring Batch: A Complete Tutorial and Example

This article provides a step‑by‑step guide on integrating Spring Batch with Spring Boot, covering business scenarios, database setup, Maven dependencies, configuration of JobRepository, JobLauncher, Job, Step, ItemReader, ItemProcessor, ItemWriter, listeners, validators, execution via REST endpoints, troubleshooting with Druid and switching to HikariCP, and demonstrates processing CSV and database data in large batches.

Batch ProcessingDatabaseJava

0 likes · 19 min read

Spring Boot Integration with Spring Batch: A Complete Tutorial and Example

Selected Java Interview Questions

Dec 28, 2022 · Backend Development

Integrating Spring Batch with Spring Boot: A Step‑by‑Step Tutorial

This article provides a comprehensive guide on using Spring Batch within a Spring Boot application, covering CSV and database data sources, detailed configuration beans, custom readers, processors, writers, listeners, validation, and execution via a REST controller, along with practical troubleshooting tips.

Batch ProcessingCSVJava

0 likes · 21 min read

Integrating Spring Batch with Spring Boot: A Step‑by‑Step Tutorial

Java Captain

Mar 21, 2025 · Backend Development

Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections

This article explains how to merge multiple user‑detail requests into a single database query using a blocking queue, scheduled thread pool, and CompletableFuture in Spring Boot, providing code examples, a high‑concurrency test, and discussion of trade‑offs such as added latency and timeout handling.

Batch ProcessingCompletableFutureConcurrency

0 likes · 13 min read

Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections

Code Ape Tech Column

Jun 4, 2025 · Backend Development

Request Collapsing Techniques: Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset

This article compares three request‑collapsing techniques—Hystrix Collapser, a custom BatchCollapser, and Guava's ConcurrentHashMultiset—explaining their designs, configurations, code implementations, and suitable scenarios for reducing downstream load in Java backend systems.

BackendBatch ProcessingConcurrency

0 likes · 13 min read

Request Collapsing Techniques: Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset

Python Programming Learning Circle

Dec 13, 2024 · Artificial Intelligence

Batch Image Translation Demo Using Youdao OCR API with Python

This article demonstrates how to build a Python desktop application that batch‑processes cosmetic product images, sends them to Youdao's OCR translation service, and displays the translated text, covering API preparation, request parameters, signature generation, and full source code.

Batch ProcessingImage TranslationOCR

0 likes · 12 min read

Batch Image Translation Demo Using Youdao OCR API with Python

Python Programming Learning Circle

Sep 3, 2021 · Fundamentals

Automating Excel with Python xlwings: Batch Create, Rename, Print, and Split Workbooks

This tutorial demonstrates how to use Python's xlwings library to programmatically create, save, list, rename, print, and split Excel workbooks and worksheets, providing practical code examples for automating common spreadsheet tasks and improving workflow efficiency.

Batch ProcessingExcel Automationpython

0 likes · 5 min read

Automating Excel with Python xlwings: Batch Create, Rename, Print, and Split Workbooks

Python Programming Learning Circle

May 12, 2020 · Artificial Intelligence

Batch Image Segmentation with Python and PaddlePaddle

This tutorial demonstrates how to use Python and the PaddlePaddle deep‑learning platform to automatically remove backgrounds from multiple photos in one step, covering installation, verification, and a concise five‑line code example for batch human segmentation.

Batch ProcessingPaddlePaddlecomputer vision

0 likes · 6 min read

Batch Image Segmentation with Python and PaddlePaddle

Lobster Programming

Apr 3, 2025 · Databases

How to Quickly Generate Batch SQL Statements Using Excel

This tutorial explains how to import data into Excel, craft formulas that concatenate cell values into UPDATE or INSERT SQL statements, and drag the formulas down to produce batch SQL commands for efficient database operations.

Batch ProcessingDatabaseSQL

0 likes · 2 min read

How to Quickly Generate Batch SQL Statements Using Excel

Java Captain

Jun 10, 2025 · Backend Development

Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide

This article explains the necessity of batch processing, presents typical use cases such as daily interest calculation, e‑commerce order archiving, log analysis and medical data migration, then dives deep into Spring Batch's core components, provides step‑by‑step code examples, performance‑tuning tips, production‑grade fault‑tolerance, monitoring solutions and a comprehensive FAQ.

Batch ProcessingJavaSpring Batch

0 likes · 20 min read

Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide

Spring Full-Stack Practical Cases

Sep 20, 2023 · Backend Development

Master Spring Batch: Build a File‑to‑Database Batch Job with Spring Boot

This tutorial walks through using Spring Batch with Spring Boot 2.6.12 to create a batch job that reads CSV data, processes each record, and writes the results into a MySQL database, covering architecture, configuration, code components, and execution steps.

Batch ProcessingJPAJava

0 likes · 11 min read

Master Spring Batch: Build a File‑to‑Database Batch Job with Spring Boot

Spring Full-Stack Practical Cases

Aug 22, 2023 · Backend Development

Master Spring Batch with Spring Boot: Build Scalable Batch Jobs Step‑by‑Step

This guide walks through setting up Spring Batch 4.2.7 with Spring Boot 2.4.12, covering architecture, configuration, job, step, reader, processor, writer, listeners, and deployment, demonstrating how to read CSV data, process it, and persist results to a MySQL database.

Batch ProcessingJPAJava

0 likes · 11 min read

Master Spring Batch with Spring Boot: Build Scalable Batch Jobs Step‑by‑Step