This guide demonstrates how to use Python's xlwings library to programmatically create, rename, list, print, and split Excel workbooks and worksheets, enabling batch processing and automation that significantly improves efficiency for repetitive spreadsheet tasks.
This article presents Tencent's practical experience of constructing a real‑time data warehouse by integrating Apache Flink with Apache Iceberg, covering background pain points of traditional Lambda architectures, Iceberg's table format and capabilities, Flink‑Iceberg sink design, small‑file handling, and future roadmap for a unified streaming‑batch data lake.
This article examines the architecture of enterprise-level big data platforms at leading Chinese tech firms—Taobao, Meituan, and Didi—detailing their data sources, synchronization components, batch and streaming processing layers, scheduling systems, and common design patterns, while highlighting shared principles across these implementations.
This article provides a comprehensive overview of Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, execution contexts, chunk processing, skip strategies, and practical tips for configuration and memory management.
This article provides a comprehensive overview of Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, practical guidelines, and how to control job execution in a Spring application.
F2 is a fast, safe Go‑based command‑line tool for batch renaming files and folders on Linux, macOS, and Windows, offering multiple installation methods, preview mode, undo, conflict resolution, built‑in variables, and powerful regex‑driven rename examples.
Spring Batch is a lightweight, comprehensive Java batch processing framework that provides reusable features such as job/step architecture, ItemReader/Writer/Processor, chunk processing, transaction management, and restart capabilities, with detailed explanations of core concepts, configuration examples, and best practices for building robust enterprise batch jobs.
This article provides a comprehensive overview of Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, and practical guidelines for building robust batch processing solutions in Java.
This article presents a collection of Python scripts for office automation, covering Word doc‑to‑docx conversion, batch address geocoding using Baidu API, distance computation with geopy, coordinate conversion between Baidu and Gaode, Excel merging, Word‑to‑PDF conversion, extracting tables from Word, and bulk Outlook email sending.
The article details Baidu’s 2020‑built commodity promotion system for Baijiahao and live‑stream e‑commerce, linking merchants with authors/streamers through CPS billing, three user interfaces and five core services, and highlights technical choices such as dynamic‑library tracking, asynchronous batch writes, and a high‑cohesion, low‑coupling architecture requiring cross‑team collaboration.
This comprehensive guide explains Spring Batch's purpose, core components such as Job, Step, ItemReader/Writer/Processor, execution flow, chunk processing, skip strategies, and practical tips for configuration, performance tuning, and troubleshooting in enterprise Java batch applications.
This article provides a comprehensive guide to Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, configuration tips, and common memory issues, all illustrated with code examples and diagrams.
This article presents a step‑by‑step Python demo that uses Youdao's OCR translation API to batch‑process cosmetic product images, covering API key setup, request parameters, signature generation, GUI implementation with Tkinter, and code snippets for file selection, result storage, and API invocation.
This article details Zhihu’s transition from a Sqoop‑driven data integration system to a Flink‑centric platform, covering business scenarios, historical architecture, design goals, technology choices, performance optimizations, and future plans for unified streaming‑batch processing across diverse storage systems.
This article details the design of a red packet calculation and distribution system for a Spring Festival promotion, addressing mixed-type packet computation, seamless transition to awarding, distributed task processing, stability guarantees, and successful implementation results.
This article explains how Apache Flink has evolved into a unified stream‑batch engine by introducing a three‑layer architecture, a unified DataStream SDK, a pipeline‑region‑based DAG scheduler, a common shuffle framework, and enhanced fault‑tolerance mechanisms to address efficiency, consistency, and resource‑utilisation challenges in real‑time big‑data processing.
This tutorial shows how to use Python's openpyxl (and optionally glob) to extract rows with purchase counts over 50 from an Excel sheet, create a new workbook, and scale the solution to batch‑process a thousand similar files automatically.
The article explains why shifting batch jobs, reporting, and machine‑learning model training from traditional backend services to a dedicated data platform can simplify development, improve fault tolerance, and scale analytics, using real‑world examples from Spotify and best‑practice guidelines.
Alibaba’s 2020 Double‑11 achieved unprecedented real‑time processing of 4 billion records per second and 7 TB of data per second using Flink, showcasing the stability, performance and efficiency of its stream‑batch unified architecture across diverse business scenarios.
This article presents Sohu Intelligent Media's data warehouse construction practice, covering fundamental concepts, batch and real‑time processing, OLAP theory, multidimensional modeling, workflow management, data quality, metadata lineage, and security, with a focus on Apache Doris and a Lambda‑style architecture.
This article introduces Alink, Alibaba's open‑source machine‑learning platform built on Flink, explains its core algorithms, performance comparison with Spark ML, version‑wise feature evolution, and provides practical quick‑start instructions for both Java (Maven) and Python (PyAlink) users, including data source handling, type conversion components, unified file‑system operations, and an overview of its FM algorithm implementation.
Learn how to use Python's openpyxl and glob modules to automatically extract rows with purchase quantities over 50 from single or thousands of Excel files, create new workbooks, and merge results, with complete code examples for both basic and advanced scenarios.
This article examines Apache Flink’s programming model, comparing its batch DataSet API with the streaming DataStream API, detailing class hierarchies, key code examples such as groupBy and job submission, and explaining how both paradigms are unified into a common JobGraph representation.
This tutorial shows how to install PaddlePaddle and PaddleHub, then use a concise Python script to batch‑remove backgrounds from multiple photos, automatically saving the segmented results, and includes sample code and before‑after images to illustrate the process.
This guide shows how to install PaddlePaddle and PaddleHub, then use a short Python script to automatically segment and remove backgrounds from multiple photos, turning each into a transparent PNG with just a few lines of code.
This tutorial demonstrates how to use Python and the PaddlePaddle deep‑learning platform to automatically remove backgrounds from multiple photos in one step, covering installation, verification, and a concise five‑line code example for batch human segmentation.
Kafka achieves its remarkable speed by combining sequential I/O, batch processing, compression, zero‑copy, careful client‑side work, and a design that avoids costly fsync and garbage collection, while maintaining durability, ordering, and at‑least‑once delivery, making it a high‑throughput, low‑latency event streaming platform.
This guide explains how to rename files on Linux using the mv and rename commands, covering basic single‑file renaming, batch renaming with patterns, handling different OS syntaxes, and advanced strategies for generating name mappings from custom files or filename patterns.
This article provides a comprehensive overview of Apache Flink, explaining how it unifies stream and batch processing on a single runtime, detailing its key features, APIs, libraries, architectural components, fault‑tolerance mechanisms, scheduling, iterative processing, and back‑pressure monitoring.
This article explains how to use SSE2 SIMD instructions to optimize string replacement operations, demonstrating a 16-character batch processing technique that significantly improves performance for longer strings.
Apache Flink 1.10 introduces a major upgrade that merges the Blink engine, boosts performance and stability, adds native Kubernetes support, enhances SQL DDL, delivers production‑ready Hive batch compatibility, optimizes memory management, and expands Python UDF capabilities, with detailed feature breakdowns and code examples.
This article presents the design of Flink's batch processing architecture, its integration with Hive through a unified Catalog API, details the enhancements in Flink 1.10, outlines future work, and reports a performance test showing roughly seven‑fold speedup over Hive on MapReduce.
This article explains how Java backend developers can replace memory‑heavy Apache POI with EasyExcel for fast, low‑memory Excel imports, covering basic reading, custom converters for gender and dates, and multiple Spring Boot integration patterns including controllers, listeners, constructor injection and lambda‑based approaches.
This article demonstrates a Java solution for automatically collecting absolute file paths, reading image URLs from a text file, and sequentially uploading over 12,000 images to a server, including code for path extraction, request handling, and error logging.
The article explains that HBase and Hive occupy distinct roles in big‑data systems—HBase handles real‑time random queries on massive detail data, while Hive provides batch‑oriented SQL‑based processing on HDFS—and describes how they are typically combined in a data pipeline.
Apache Flink 1.9.0, released on August 22, merges Alibaba's Blink engine, introduces a major architecture overhaul, enriches Table API & SQL, adds batch and stream processing enhancements, and integrates tightly with Hive, marking a significant milestone for large‑scale data processing.
FeatureKV, WeChat’s high‑performance key‑value store, handles one‑billion queries per second and ingests a billion keys per hour by separating write‑only DataSvr from read‑only KVSvr, supporting in‑memory, indexed, and block‑indexed tables, scaling horizontally, guaranteeing eventual consistency with versioned reads, and delivering up to 11 billion reads per second with sub‑15 ms latency.
This tutorial walks you through building a simple Spring Batch Hello World job using Spring Boot and Maven, covering the framework basics, project setup, Maven configuration, entity modeling, job and step configuration, data processing, testing, and how to run the example to generate greeting output.
This article explains the technical challenges of large‑scale data processing, compares the classic Lambda and Kappa architectures, and introduces the cloud‑native Lambda Plus solution built on TableStore and Blink that simplifies batch‑stream integration for TB‑scale workloads.
This article chronicles a step‑by‑step performance overhaul of a 20‑million‑record migration project, detailing four architectural revisions—from a single‑threaded procedural script to a fully decoupled, multithreaded, interface‑driven solution—that reduced processing time from two months to just four hours while ensuring data consistency, recoverability, and scalability.
This guide provides a Python script that automates the batch conversion of Microsoft PowerPoint files (.ppt or .pptx) to PDF, including setup steps, full source code, and instructions for running the script on a folder of presentations.
This article explains Apache Flink’s fundamental design principle that treats batch as a special case of stream, compares native streaming with micro‑batching, describes its deployment modes, fault‑tolerance mechanisms, unified data and scheduling layers, and outlines Alibaba’s architectural optimizations for the platform.
Alibaba’s Blink, an internal enhancement of Apache Flink, is now open‑sourced, bringing advanced runtime, SQL/TableAPI, Hive compatibility, Zeppelin integration, and a revamped Flink Web UI to dramatically boost performance and scalability for both streaming and batch workloads.
This article examines TiDB, a world‑class open‑source distributed NewSQL database, comparing its transaction and query behavior with MySQL, discussing underlying Percolator model, server‑side prepared statements, batch processing techniques, and practical optimization strategies for developers.
Learn how to quickly rename multiple files in Linux using the rename utility, a bash for‑loop with mv, and a sed‑based loop, each illustrated with clear command examples and explanations of the underlying string manipulation techniques.
The article introduces a comparative test platform for financial core systems that automates regression testing of batch jobs, dramatically reducing test cycles from days to under one, while ensuring billing calculations remain accurate across evolving business scenarios.
This article introduces Apache Flink’s rapid rise as the leading open‑source big data engine, explains its role in batch, stream, and interactive analytics, showcases real‑world use cases from Alibaba, Didi, and ByteDance, and outlines how Flink powers both big data and AI workloads.
After struggling with a two‑month data migration that processed 20 million users, the author iteratively redesigned the system—from a single‑threaded, procedural approach to a fully decoupled, multithreaded architecture using queues and batch operations—ultimately reducing runtime to just four hours while ensuring data consistency and recoverability.
The article examines Hadoop's relevance by contrasting it with PostgreSQL and Storm, discussing when each technology fits big‑data challenges such as volume, velocity, and variety, and highlighting cost, complexity, and use‑case considerations for enterprises.
This article details the design of a unified MySQL data synchronization platform that consolidates offline sync, real‑time subscription, and real‑time sync into BatchJob, StreamJob, and PieJob abstractions, describing task implementations, cluster architecture, high‑availability mechanisms, and evolution challenges such as file loss and metadata handling.
This article examines the evolution of high‑performance computing from exclusive national supercomputers to accessible cloud‑based services, outlines the three adoption stages for enterprises, and details Tencent Cloud's Batch platform features that simplify large‑scale HPC workloads.
This article introduces the evolution of big‑data processing from Google’s MapReduce concept to modern open‑source frameworks, defines big data and its 3V characteristics, outlines typical processing pipelines, and compares batch, stream, and hybrid systems such as Hadoop, Storm, Samza, Spark, and Flink.
This article shares practical patterns and built‑in mechanisms for dealing with duplicate messages, message ordering, concurrent updates, asynchronous acknowledgments, and batch processing in a large‑scale, message‑driven architecture, illustrated with QMQ examples from Qunar's platform.
This article shares Qunar's practical experiences with message-driven architecture, detailing consumer handling of duplicate messages, ordering, concurrency control, asynchronous processing, and batch strategies, and presents concrete solutions such as idempotent checks, deduplication tables, versioning, and QMQ's built‑in executors.
This article introduces Apache Beam, its Beam Model, and how the Beam SDK enables developers to write unified, flexible pipelines for both bounded batch jobs and unbounded streaming workloads, illustrating concepts with mobile‑gaming examples and detailed code snippets.
This article explains the principles of Lambda Architecture, its three‑layer design for combining batch and real‑time analytics, and demonstrates a detailed smart‑traffic case study with component selection, capacity planning, and implementation guidance for building scalable big‑data systems.
This guide explains how to clean redundant fields, restructure a massive MySQL table, batch‑fetch data with LIMIT, rewrite slow queries, and compare three insertion methods—including prepared statements and bulk inserts—to cut migration time from hours to under 20 minutes.
This article explains the classic Oracle batch‑processing issue caused by inaccurate statistics on tables with sharply changing row counts, reviews traditional mitigation methods and their drawbacks, and introduces Oracle 12c's online statistics gathering for bulk‑load as an automated solution while noting its current limitations.
Apache Spark, a versatile big‑data platform supporting batch processing, SQL queries, real‑time streaming, and machine‑learning workloads, dramatically accelerates data‑intensive jobs, as demonstrated by Meituan‑Dianping, where its high‑performance engine reduces execution times and enhances scalability across diverse analytical and operational pipelines.
This article explains the purpose of the Unix xargs command, demonstrates basic usage for concatenating log files, shows an advanced example for renaming text files to log files using the -I placeholder, and provides a step‑by‑step breakdown of how the command pipeline works.
