The article explains how to merge multiple user queries into a single batch request using a queue, scheduled thread pool and CompletableFuture in Spring Boot, thereby reducing database connections and improving resource utilization while handling high‑concurrency scenarios.
This article summarizes practical techniques for reducing API latency, including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction size control, program structure refactoring, deep pagination handling, SQL tuning, and proper lock granularity.
This article compares three request‑merging approaches—Hystrix Collapser, a custom BatchCollapser implementation, and Guava’s ConcurrentHashMultiset—explaining their configurations, code examples, and suitable scenarios for reducing downstream load and improving system throughput while also highlighting performance trade‑offs and practical tips for integration in Spring‑Boot services.
This article presents a complete guide to Hudi-based real-time data lake ingestion, covering overall data integration architecture, batch and streaming ingestion strategies, advanced table design, and practical recommendations for handling challenges such as deduplication, latency, partitioning, and performance optimization.
This article presents a comprehensive overview of evolving big‑data architectures—from classic offline warehouses to Lambda and Kappa models—and details a streaming‑batch integrated solution that addresses latency, data freshness, and multi‑table join challenges to achieve minute‑level real‑time multi‑dimensional analytics.
This article provides a comprehensive overview of Apache SeaTunnel, covering its design objectives, current capabilities such as multi‑engine support and extensive connector ecosystem, detailed architecture including engine‑independent APIs and execution flows, and outlines the upcoming roadmap to expand connectors, launch a visual web UI, and introduce a dedicated SeaTunnel Engine.
This article presents Akulaku’s practical experience with OpenMLDB, describing the company’s data‑driven requirements, the design of a unified stream‑batch architecture, implementation details across offline, online and RocksDB modes, and future recommendations for high‑performance, scenario‑agnostic big‑data processing.
This article introduces Alink, Alibaba's open‑source machine‑learning platform built on Flink, explains its core algorithms, performance comparison with Spark ML, version‑wise feature evolution, and provides practical quick‑start instructions for both Java (Maven) and Python (PyAlink) users, including data source handling, type conversion components, unified file‑system operations, and an overview of its FM algorithm implementation.
This article presents the design of Flink's batch processing architecture, its integration with Hive through a unified Catalog API, details the enhancements in Flink 1.10, outlines future work, and reports a performance test showing roughly seven‑fold speedup over Hive on MapReduce.
This article experimentally compares three Java‑based insertion strategies—MyBatis without transaction, plain JDBC with and without transaction, and JDBC batch processing—with and without transactions, measuring how they affect the time required to insert tens of millions of rows into MySQL.
This article explains how designers and developers can use a short Python script to automatically compress large numbers of high‑resolution images, covering the problem background, required environment, step‑by‑step implementation, and the impressive compression results achieved.
This article presents a collection of Python scripts for office automation, covering Word doc‑to‑docx conversion, batch address geocoding using Baidu API, distance computation with geopy, coordinate conversion between Baidu and Gaode, Excel merging, Word‑to‑PDF conversion, extracting tables from Word, and bulk Outlook email sending.
This article presents a step‑by‑step Python demo that uses Youdao's OCR translation API to batch‑process cosmetic product images, covering API key setup, request parameters, signature generation, GUI implementation with Tkinter, and code snippets for file selection, result storage, and API invocation.
The article introduces a comparative test platform for financial core systems that automates regression testing of batch jobs, dramatically reducing test cycles from days to under one, while ensuring billing calculations remain accurate across evolving business scenarios.
This article explores a real‑world scenario where a support manager faces nightly batch job interruptions, analyzes common database and environment failures, and presents a systematic redesign of the batch framework and executor to enable automatic retry, reducing manual intervention and improving operational reliability.
At KubeCon+CloudNativeCon 2023 in Amsterdam, Volcano Engine and ByteDance presented two technical sessions covering Dragonfly's P2P image distribution best practices and large‑scale Kubernetes batch processing strategies, offering deep insights and real‑world implementations for cloud‑native developers.
This article shares Qunar's practical experiences with message-driven architecture, detailing consumer handling of duplicate messages, ordering, concurrency control, asynchronous processing, and batch strategies, and presents concrete solutions such as idempotent checks, deduplication tables, versioning, and QMQ's built‑in executors.
The article explores a wide range of performance‑optimization strategies—including indexing, caching, compression, prefetching, peak‑shaving, and batch processing—explaining their trade‑offs, practical applications, and how they relate to hardware latency and system design in modern computing environments.
This article presents a thorough, multi‑part exploration of software performance optimization techniques—including indexing, compression, caching, prefetching, peak‑shaving, and batch processing—explaining their principles, trade‑offs, practical applications, and how they relate to hardware constraints and system design.
This article explains how to build a unified stream‑batch real‑time data warehouse using FlinkSQL, covering prerequisite knowledge, five core concepts, two implementation approaches, a comparison of traditional versus real‑time architectures, and a comprehensive hands‑on example, illustrated with diagrams.