BestHub
Sign in
Tagged articles

Batch Processing

251 articles · Page 3 of 3
Baidu Geek Talk
Baidu Geek Talk
Jul 7, 2021 · Backend Development

Design and Implementation of Baidu's Commodity Promotion System

The article details Baidu’s 2020‑built commodity promotion system for Baijiahao and live‑stream e‑commerce, linking merchants with authors/streamers through CPS billing, three user interfaces and five core services, and highlights technical choices such as dynamic‑library tracking, asynchronous batch writes, and a high‑cohesion, low‑coupling architecture requiring cross‑team collaboration.

Baidu ecosystemBatch ProcessingCPS
0 likes · 15 min read
Design and Implementation of Baidu's Commodity Promotion System
Programmer DD
Programmer DD
Jul 5, 2021 · Backend Development

Master Spring Batch: Core Concepts, Architecture, and Best Practices

This comprehensive guide explains Spring Batch's purpose, core components such as Job, Step, ItemReader/Writer/Processor, execution flow, chunk processing, skip strategies, and practical tips for configuration, performance tuning, and troubleshooting in enterprise Java batch applications.

Batch ProcessingChunkJob
0 likes · 19 min read
Master Spring Batch: Core Concepts, Architecture, and Best Practices
Code Ape Tech Column
Code Ape Tech Column
Jul 1, 2021 · Backend Development

Master Spring Batch: Core Concepts, Architecture, and Practical Tips

This article provides a comprehensive guide to Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, configuration tips, and common memory issues, all illustrated with code examples and diagrams.

Batch ProcessingChunk ProcessingData Integration
0 likes · 19 min read
Master Spring Batch: Core Concepts, Architecture, and Practical Tips
Python Programming Learning Circle
Python Programming Learning Circle
Jun 25, 2021 · Artificial Intelligence

Batch Image Translation Demo Using Youdao OCR API with Python

This article presents a step‑by‑step Python demo that uses Youdao's OCR translation API to batch‑process cosmetic product images, covering API key setup, request parameters, signature generation, GUI implementation with Tkinter, and code snippets for file selection, result storage, and API invocation.

AIBatch ProcessingOCR
0 likes · 10 min read
Batch Image Translation Demo Using Youdao OCR API with Python
DataFunTalk
DataFunTalk
Apr 23, 2021 · Big Data

Building and Evolving Zhihu’s Flink‑Based Data Integration Platform

This article details Zhihu’s transition from a Sqoop‑driven data integration system to a Flink‑centric platform, covering business scenarios, historical architecture, design goals, technology choices, performance optimizations, and future plans for unified streaming‑batch processing across diverse storage systems.

Batch ProcessingBig DataData Integration
0 likes · 14 min read
Building and Evolving Zhihu’s Flink‑Based Data Integration Platform
Kuaishou Tech
Kuaishou Tech
Apr 9, 2021 · Backend Development

Design and Implementation of Red Packet Calculation and Distribution System for Spring Festival Activity

This article details the design of a red packet calculation and distribution system for a Spring Festival promotion, addressing mixed-type packet computation, seamless transition to awarding, distributed task processing, stability guarantees, and successful implementation results.

Batch ProcessingDistributed ComputingFault Tolerance
0 likes · 12 min read
Design and Implementation of Red Packet Calculation and Distribution System for Spring Festival Activity
DataFunTalk
DataFunTalk
Mar 28, 2021 · Big Data

Flink Stream‑Batch Integration: Layered Architecture, Unified SDK, DAG Scheduler, Shuffle, and Fault‑Tolerance

This article explains how Apache Flink has evolved into a unified stream‑batch engine by introducing a three‑layer architecture, a unified DataStream SDK, a pipeline‑region‑based DAG scheduler, a common shuffle framework, and enhanced fault‑tolerance mechanisms to address efficiency, consistency, and resource‑utilisation challenges in real‑time big‑data processing.

Apache FlinkBatch ProcessingDAG scheduler
0 likes · 25 min read
Flink Stream‑Batch Integration: Layered Architecture, Unified SDK, DAG Scheduler, Shuffle, and Fault‑Tolerance
DataFunSummit
DataFunSummit
Nov 17, 2020 · Big Data

Sohu Intelligent Media Data Warehouse Architecture and Technical Practices

This article presents Sohu Intelligent Media's data warehouse construction practice, covering fundamental concepts, batch and real‑time processing, OLAP theory, multidimensional modeling, workflow management, data quality, metadata lineage, and security, with a focus on Apache Doris and a Lambda‑style architecture.

Apache DorisBatch ProcessingData Warehouse
0 likes · 18 min read
Sohu Intelligent Media Data Warehouse Architecture and Technical Practices
DataFunTalk
DataFunTalk
Nov 17, 2020 · Artificial Intelligence

Alink: A Flink‑Based Machine Learning Platform – Overview, Features, and Quick‑Start Guide

This article introduces Alink, Alibaba's open‑source machine‑learning platform built on Flink, explains its core algorithms, performance comparison with Spark ML, version‑wise feature evolution, and provides practical quick‑start instructions for both Java (Maven) and Python (PyAlink) users, including data source handling, type conversion components, unified file‑system operations, and an overview of its FM algorithm implementation.

AlinkBatch ProcessingData Integration
0 likes · 13 min read
Alink: A Flink‑Based Machine Learning Platform – Overview, Features, and Quick‑Start Guide
Architect
Architect
May 30, 2020 · Big Data

Understanding Flink’s Unified Programming API for Batch and Streaming Jobs

This article examines Apache Flink’s programming model, comparing its batch DataSet API with the streaming DataStream API, detailing class hierarchies, key code examples such as groupBy and job submission, and explaining how both paradigms are unified into a common JobGraph representation.

Batch ProcessingBig DataFlink
0 likes · 9 min read
Understanding Flink’s Unified Programming API for Batch and Streaming Jobs
21CTO
21CTO
May 20, 2020 · Artificial Intelligence

Batch Remove Image Backgrounds with Python and PaddlePaddle

This tutorial shows how to install PaddlePaddle and PaddleHub, then use a concise Python script to batch‑remove backgrounds from multiple photos, automatically saving the segmented results, and includes sample code and before‑after images to illustrate the process.

Batch ProcessingPaddlePaddlePython
0 likes · 5 min read
Batch Remove Image Backgrounds with Python and PaddlePaddle
ITPUB
ITPUB
May 15, 2020 · Artificial Intelligence

Batch Background Removal with PaddlePaddle: One-Click Python Script

This guide shows how to install PaddlePaddle and PaddleHub, then use a short Python script to automatically segment and remove backgrounds from multiple photos, turning each into a transparent PNG with just a few lines of code.

Batch ProcessingPaddlePaddleimage segmentation
0 likes · 5 min read
Batch Background Removal with PaddlePaddle: One-Click Python Script
Python Programming Learning Circle
Python Programming Learning Circle
May 12, 2020 · Artificial Intelligence

Batch Image Segmentation with Python and PaddlePaddle

This tutorial demonstrates how to use Python and the PaddlePaddle deep‑learning platform to automatically remove backgrounds from multiple photos in one step, covering installation, verification, and a concise five‑line code example for batch human segmentation.

Batch ProcessingPaddlePaddlecomputer vision
0 likes · 6 min read
Batch Image Segmentation with Python and PaddlePaddle
Programmer DD
Programmer DD
Mar 28, 2020 · Backend Development

Why Is Kafka So Fast? Uncover the 11 Performance Secrets

Kafka achieves its remarkable speed by combining sequential I/O, batch processing, compression, zero‑copy, careful client‑side work, and a design that avoids costly fsync and garbage collection, while maintaining durability, ordering, and at‑least‑once delivery, making it a high‑throughput, low‑latency event streaming platform.

Batch ProcessingKafkaMessage Queue
0 likes · 15 min read
Why Is Kafka So Fast? Uncover the 11 Performance Secrets
Liangxu Linux
Liangxu Linux
Mar 28, 2020 · Fundamentals

Master Linux File Renaming with mv and rename: Simple and Complex Techniques

This guide explains how to rename files on Linux using the mv and rename commands, covering basic single‑file renaming, batch renaming with patterns, handling different OS syntaxes, and advanced strategies for generating name mappings from custom files or filename patterns.

Batch ProcessingShell scriptingawk
0 likes · 11 min read
Master Linux File Renaming with mv and rename: Simple and Complex Techniques
Architecture Digest
Architecture Digest
Mar 11, 2020 · Big Data

Apache Flink: Unified Stream and Batch Processing Architecture and Core Concepts

This article provides a comprehensive overview of Apache Flink, explaining how it unifies stream and batch processing on a single runtime, detailing its key features, APIs, libraries, architectural components, fault‑tolerance mechanisms, scheduling, iterative processing, and back‑pressure monitoring.

Apache FlinkBatch ProcessingDistributed Computing
0 likes · 20 min read
Apache Flink: Unified Stream and Batch Processing Architecture and Core Concepts
Alibaba Cloud Developer
Alibaba Cloud Developer
Feb 24, 2020 · Big Data

What’s New in Apache Flink 1.10? Deep Dive into Major Features and Enhancements

Apache Flink 1.10 introduces a major upgrade that merges the Blink engine, boosts performance and stability, adds native Kubernetes support, enhances SQL DDL, delivers production‑ready Hive batch compatibility, optimizes memory management, and expands Python UDF capabilities, with detailed feature breakdowns and code examples.

Apache FlinkBatch ProcessingKubernetes
0 likes · 8 min read
What’s New in Apache Flink 1.10? Deep Dive into Major Features and Enhancements
Programmer DD
Programmer DD
Nov 4, 2019 · Backend Development

Master Efficient Excel Imports in Java with EasyExcel and Spring Boot

This article explains how Java backend developers can replace memory‑heavy Apache POI with EasyExcel for fast, low‑memory Excel imports, covering basic reading, custom converters for gender and dates, and multiple Spring Boot integration patterns including controllers, listeners, constructor injection and lambda‑based approaches.

Batch ProcessingEasyExcelExcel
0 likes · 12 min read
Master Efficient Excel Imports in Java with EasyExcel and Spring Boot
Tencent Cloud Developer
Tencent Cloud Developer
Aug 1, 2019 · Databases

FeatureKV: A High-Performance Key-Value Storage System for WeChat's Billion-Scale Challenges

FeatureKV, WeChat’s high‑performance key‑value store, handles one‑billion queries per second and ingests a billion keys per hour by separating write‑only DataSvr from read‑only KVSvr, supporting in‑memory, indexed, and block‑indexed tables, scaling horizontally, guaranteeing eventual consistency with versioned reads, and delivering up to 11 billion reads per second with sub‑15 ms latency.

Batch ProcessingFeatureKVKey-Value
0 likes · 22 min read
FeatureKV: A High-Performance Key-Value Storage System for WeChat's Billion-Scale Challenges
Programmer DD
Programmer DD
Jul 14, 2019 · Backend Development

Step‑by‑Step Spring Batch Hello World Job with Spring Boot

This tutorial walks you through building a simple Spring Batch Hello World job using Spring Boot and Maven, covering the framework basics, project setup, Maven configuration, entity modeling, job and step configuration, data processing, testing, and how to run the example to generate greeting output.

Batch ProcessingSpring Batch
0 likes · 19 min read
Step‑by‑Step Spring Batch Hello World Job with Spring Boot
Qunar Tech Salon
Qunar Tech Salon
Jul 5, 2019 · Big Data

Understanding Big Data Processing Architectures: Lambda, Kappa, and Lambda Plus

This article explains the technical challenges of large‑scale data processing, compares the classic Lambda and Kappa architectures, and introduces the cloud‑native Lambda Plus solution built on TableStore and Blink that simplifies batch‑stream integration for TB‑scale workloads.

Batch ProcessingKappa ArchitectureLambda Architecture
0 likes · 13 min read
Understanding Big Data Processing Architectures: Lambda, Kappa, and Lambda Plus
Programmer DD
Programmer DD
Jun 16, 2019 · Backend Development

From Two‑Month Crawl to Four‑Hour Sprint: Optimizing a 20M‑Record Data Migration

This article chronicles a step‑by‑step performance overhaul of a 20‑million‑record migration project, detailing four architectural revisions—from a single‑threaded procedural script to a fully decoupled, multithreaded, interface‑driven solution—that reduced processing time from two months to just four hours while ensuring data consistency, recoverability, and scalability.

Batch ProcessingData MigrationSpring Boot
0 likes · 15 min read
From Two‑Month Crawl to Four‑Hour Sprint: Optimizing a 20M‑Record Data Migration
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 12, 2019 · Big Data

Understanding Apache Flink’s Core Design: “Batch Is a Special Case of Stream” and Its Architecture

This article explains Apache Flink’s fundamental design principle that treats batch as a special case of stream, compares native streaming with micro‑batching, describes its deployment modes, fault‑tolerance mechanisms, unified data and scheduling layers, and outlines Alibaba’s architectural optimizations for the platform.

Apache FlinkBatch ProcessingStream Processing
0 likes · 15 min read
Understanding Apache Flink’s Core Design: “Batch Is a Special Case of Stream” and Its Architecture
Zhuanzhuan Tech
Zhuanzhuan Tech
Jan 11, 2019 · Databases

Differences Between TiDB and MySQL: Transactions, Queries, Server‑Side Prepared Statements, and Batch Processing

This article examines TiDB, a world‑class open‑source distributed NewSQL database, comparing its transaction and query behavior with MySQL, discussing underlying Percolator model, server‑side prepared statements, batch processing techniques, and practical optimization strategies for developers.

Batch ProcessingMySQLQuery Optimization
0 likes · 10 min read
Differences Between TiDB and MySQL: Transactions, Queries, Server‑Side Prepared Statements, and Batch Processing
Liangxu Linux
Liangxu Linux
Nov 29, 2018 · Fundamentals

Three Simple Linux Commands to Batch Rename Files Efficiently

Learn how to quickly rename multiple files in Linux using the rename utility, a bash for‑loop with mv, and a sed‑based loop, each illustrated with clear command examples and explanations of the underlying string manipulation techniques.

Batch ProcessingFile Renamingbash
0 likes · 5 min read
Three Simple Linux Commands to Batch Rename Files Efficiently
Java Backend Technology
Java Backend Technology
Jul 29, 2018 · Backend Development

Turning a 2‑Month Data Migration into 4 Hours: Backend Performance Secrets

After struggling with a two‑month data migration that processed 20 million users, the author iteratively redesigned the system—from a single‑threaded, procedural approach to a fully decoupled, multithreaded architecture using queues and batch operations—ultimately reducing runtime to just four hours while ensuring data consistency and recoverability.

Batch ProcessingData Migrationmultithreading
0 likes · 13 min read
Turning a 2‑Month Data Migration into 4 Hours: Backend Performance Secrets
ITPUB
ITPUB
Jun 19, 2018 · Big Data

Is Hadoop Still Relevant? Comparing Hadoop, PostgreSQL, and Storm

The article examines Hadoop's relevance by contrasting it with PostgreSQL and Storm, discussing when each technology fits big‑data challenges such as volume, velocity, and variety, and highlighting cost, complexity, and use‑case considerations for enterprises.

Batch ProcessingHadoopPostgreSQL
0 likes · 8 min read
Is Hadoop Still Relevant? Comparing Hadoop, PostgreSQL, and Storm
JD Tech
JD Tech
May 31, 2018 · Backend Development

Design and Architecture of a Unified MySQL Data Synchronization Platform

This article details the design of a unified MySQL data synchronization platform that consolidates offline sync, real‑time subscription, and real‑time sync into BatchJob, StreamJob, and PieJob abstractions, describing task implementations, cluster architecture, high‑availability mechanisms, and evolution challenges such as file loss and metadata handling.

Backend ArchitectureBatch ProcessingData synchronization
0 likes · 10 min read
Design and Architecture of a Unified MySQL Data Synchronization Platform
Ctrip Technology
Ctrip Technology
Jan 25, 2017 · Backend Development

Handling Duplicate Messages, Ordering, Concurrency, and Batch Processing in Message‑Driven Systems

This article shares practical patterns and built‑in mechanisms for dealing with duplicate messages, message ordering, concurrent updates, asynchronous acknowledgments, and batch processing in a large‑scale, message‑driven architecture, illustrated with QMQ examples from Qunar's platform.

Batch ProcessingConcurrency ControlDuplicate Message Handling
0 likes · 16 min read
Handling Duplicate Messages, Ordering, Concurrency, and Batch Processing in Message‑Driven Systems
Qunar Tech Salon
Qunar Tech Salon
Jan 21, 2017 · Backend Development

Message Consumption Patterns and Best Practices in Qunar's QMQ

This article shares Qunar's practical experiences with message-driven architecture, detailing consumer handling of duplicate messages, ordering, concurrency control, asynchronous processing, and batch strategies, and presents concrete solutions such as idempotent checks, deduplication tables, versioning, and QMQ's built‑in executors.

Batch ProcessingOrderingasynchronous ack
0 likes · 18 min read
Message Consumption Patterns and Best Practices in Qunar's QMQ
StarRing Big Data Open Lab
StarRing Big Data Open Lab
Oct 10, 2016 · Big Data

Mastering Lambda Architecture: Real‑Time & Batch Processing for Smart Traffic

This article explains the principles of Lambda Architecture, its three‑layer design for combining batch and real‑time analytics, and demonstrates a detailed smart‑traffic case study with component selection, capacity planning, and implementation guidance for building scalable big‑data systems.

Batch ProcessingLambda ArchitectureSmart Traffic
0 likes · 15 min read
Mastering Lambda Architecture: Real‑Time & Batch Processing for Smart Traffic
ITPUB
ITPUB
Sep 7, 2016 · Databases

Optimizing and Migrating a 20 Million‑Row MySQL Table

This guide explains how to clean redundant fields, restructure a massive MySQL table, batch‑fetch data with LIMIT, rewrite slow queries, and compare three insertion methods—including prepared statements and bulk inserts—to cut migration time from hours to under 20 minutes.

Batch ProcessingData MigrationMySQL
0 likes · 7 min read
Optimizing and Migrating a 20 Million‑Row MySQL Table
dbaplus Community
dbaplus Community
Aug 10, 2016 · Databases

How Oracle 12c Solves Batch‑Load Statistics Problems with Online Gathering

This article explains the classic Oracle batch‑processing issue caused by inaccurate statistics on tables with sharply changing row counts, reviews traditional mitigation methods and their drawbacks, and introduces Oracle 12c's online statistics gathering for bulk‑load as an automated solution while noting its current limitations.

12cBatch ProcessingOracle
0 likes · 9 min read
How Oracle 12c Solves Batch‑Load Statistics Problems with Online Gathering
Meituan Technology Team
Meituan Technology Team
Apr 29, 2016 · Big Data

Introduction to Spark in Big Data

Apache Spark, a versatile big‑data platform supporting batch processing, SQL queries, real‑time streaming, and machine‑learning workloads, dramatically accelerates data‑intensive jobs, as demonstrated by Meituan‑Dianping, where its high‑performance engine reduces execution times and enhances scalability across diverse analytical and operational pipelines.

Batch ProcessingBig DataSpark
0 likes · 1 min read
Introduction to Spark in Big Data
Qunar Tech Salon
Qunar Tech Salon
Jul 28, 2015 · Fundamentals

Introduction to xargs with Basic and Advanced Usage Examples

This article explains the purpose of the Unix xargs command, demonstrates basic usage for concatenating log files, shows an advanced example for renaming text files to log files using the -I placeholder, and provides a step‑by‑step breakdown of how the command pipeline works.

Batch ProcessingScriptingUnix
0 likes · 3 min read
Introduction to xargs with Basic and Advanced Usage Examples