A graduate student and his lab used Baidu Comate, an AI‑powered coding assistant, to automate repetitive Python scripts for converting 150 GB of DICOM images to PNG, reducing a week‑long, three‑person effort to two days for a single developer and boosting overall team efficiency.
This article details the necessity of establishing a no-buried-point scheme mindset, key advantages over traditional methods, mobile client no-buried-point collection implementation mechanisms, data processing, and the benefits of this approach.
This article demonstrates how to build a high‑performance data processing engine using PHP and SOAP by setting up a SOAP server, creating a client, and applying optimization techniques such as efficient function design, caching, asynchronous processing, and database tuning.
This article compares Pandas, Polars, and PySpark across five dataset sizes, showing how Polars' eager and lazy modes dramatically outperform the other tools, and discusses when each framework is the most suitable choice for data processing workloads.
This article explains how Laravel's chunk method can process massive user datasets in small batches, reducing memory usage and improving performance, with example code for sending welcome emails and controlling flow using return false.
This article presents a Python project that reads point‑cloud data from CSV files, applies DBSCAN and KMeans clustering, extracts cluster features, and uses the Hungarian algorithm to match clusters across frames for tracking, complete with full source code and result visualization.
This article explains Python’s built‑in map() function, outlines common scenarios such as type conversion, numeric calculations, and preprocessing, and provides ten concrete code examples that demonstrate how to apply map() for data transformation, mathematical operations, and custom logic.
The talk explores how the rapid growth of multimodal data and large language models is reshaping data processing, highlighting three key trends—online‑offline integration, vector‑relational database convergence, and the fusion of data processing with AI computation—while presenting practical solutions and future visions for unified data‑AI ecosystems.
This article explains the logical execution order of SQL statements—including FROM‑JOIN, WHERE, GROUP BY, HAVING, SELECT, DISTINCT, ORDER BY and LIMIT—illustrates each step with examples and diagrams, and clarifies common misconceptions about clause sequencing.
This article walks through nine Go‑based implementations for the 1‑Billion‑Row Challenge, starting from a straightforward scanner approach and progressively applying map pointer values, custom parsing, integer arithmetic, buffer tweaks, custom hash tables, and parallelism to shrink processing time from 1 minute 45 seconds to under 4 seconds.
This article presents a comprehensive overview of Vivo's BlueLM large language model, covering its historical evolution, training pipeline and data challenges, algorithmic innovations, safety measures, edge‑device optimization, product deployments such as BlueLM Mini‑V and BlueQianXun, and insights from a detailed Q&A session.
This article walks through using Python and pandas to batch‑process seven Excel evaluation sheets, skipping header rows, cleaning data, computing total and average scores per person, merging results, and outputting aggregated statistics, providing a practical automation solution for repetitive office tasks.
This article walks through handling merged headers and formula cells in Excel using Python's pandas and openpyxl, providing step‑by‑step code to merge multiple workbooks, skip unwanted rows, assign custom headers, and format the final sheet with merged cells.
This article introduces Java 8's Stream API, explains its pipeline concept similar to SQL and Linux pipes, and demonstrates common operations such as filter, map, sorted, forEach, collect, statistics and parallelStream using a UserPo class with complete runnable code examples.
This article walks through a Python automation solution that extracts contract names from Excel using pandas and regex, cleans the data, and then matches and moves the corresponding contract files to a target directory, complete with code examples and results.
This article presents a comprehensive overview of Vivo's BlueLM large language model, covering its historical evolution, the massive data and algorithmic challenges faced during training, safety and performance optimizations, and how the model has been integrated into various consumer and enterprise products.
This article walks through a practical Pandas solution for merging several text files in a folder, showing how to read each file, concatenate the data frames, and export the combined result as a CSV, complete with sample code and usage tips.
This article presents 17 practical Python scripts covering file management, web scraping, email handling, Excel processing, database interaction, system tasks, image editing, and more, each with code examples and explanations, enabling developers and analysts to automate repetitive workflows and boost productivity across diverse domains.
This guide explains Logstash’s Grok filter plugin, detailing how its 120 built‑in and custom patterns transform unstructured logs—such as Apache, MySQL, or HiveServer2—into structured fields through named regex captures, supporting type conversion, cleaning, debugging, and efficient ETL for analysis and monitoring.
This article explains the 1 Billion Row Challenge (1BRC) for Java 21, detailing the data format, required output, how to build and run the benchmark, optimization options, submission rules, and the evaluation environment used to rank participants.
This article walks through a Python automation challenge where records are grouped by several fields, sorted by end time, and only the first entry is retained when consecutive timestamps differ by 20 seconds or less, providing clear code and visual results.
This article shows how flexible, divergent thinking combined with basic MaxCompute (ODPS) SQL syntax can solve complex data problems such as generating sequences, splitting intervals, performing permutations and combinations, and analyzing continuous activity, providing step‑by‑step examples, SQL code snippets, and practical results.
esProc SPL, a pure‑Java, lightweight data‑processing engine, offers richer data source support, built‑in flow control, and easier complex calculations compared to SQLite, making it a powerful alternative for small applications that need database‑like capabilities without the overhead of traditional databases.
The article describes a frontend error‑message governance system that dynamically maps cryptic error codes to clear, context‑aware messages, lets operators configure responses by code, URL or endpoint, and improves user experience and system stability while enabling future features such as efficiency metrics, guides, and surveys.
This article walks through a Pandas data‑processing question about rounding loan‑balance sums to two decimal places, provides a working code snippet, and also announces free e‑book giveaways and community groups for Python learners.
This article walks through handling Excel columns that contain both compact (YYYYMMDD) and full timestamp (YYYY-MM-DD HH:MM:SS) date strings in Pandas, showing how to unify formats by removing hyphens, trimming, and converting them to proper datetime objects with concise code.
This article walks through a Python‑based solution for copying rows from an Excel sheet into specific cells of another workbook and generating Word documents, showcasing the problem, a detailed code example, common pitfalls, and a concise summary of the approach.
This article demonstrates two PHP approaches—manual looping and functional array utilities—to group and count records in a two‑dimensional array based on identical name and age fields, showing the full source code and expected output.
This article walks through optimizing a Python script that reads GPS data from a text file, extracts latitude‑longitude pairs using regular expressions, splits them into separate columns with pandas, and saves the cleaned results to Excel, while offering practical tips and code snippets for efficient data handling.
This article walks through using Excel’s import, text‑to‑columns, and find‑replace features to restructure raw data into a clean format suitable for Python automation, illustrating each step with screenshots and summarizing the final result.
This tutorial demonstrates how to use the Python openpyxl library to create, open, edit, and save Excel workbooks, covering worksheets, cells, ranges, formulas, merging, iteration, and file handling for both small and large spreadsheets.
Learn how to automate the consolidation of numerous employee performance Excel files using Python and pandas, with a clear code example that reads each workbook, concatenates the data, and outputs a single merged spreadsheet, while also offering tips for handling file formats and avoiding common pitfalls.
This comprehensive tutorial explains Java 8's Stream API and Lambda expressions, demonstrating how to create streams from collections and arrays, perform intermediate operations like filter, map, flatMap, and sorted, and use terminal operations such as forEach, findFirst, reduce, and collect through twenty detailed code examples covering employee data processing, aggregation, grouping, and sorting.
This guide explains how to use pandas together with the xlsxwriter engine to create a DataFrame, embed HYPERLINK formulas for URLs, files, and images, format them, and write the result to an Excel workbook, including installation steps and multiple code examples.
This article demonstrates how to use Python and pandas to automate the extraction, filtering, and handling of contract data from Excel files, replacing manual per‑contract spreadsheets with a single reusable script that improves efficiency and reduces errors.
The article analyzes SQLite's shortcomings in data‑source support, complex calculations, and workflow handling for tiny Java apps, then demonstrates how the open‑source esProc SPL engine offers richer data‑source integration, simpler SQL‑like syntax, powerful calculation capabilities, and built‑in flow control, making it a more suitable lightweight database alternative.
This article walks through automating the aggregation of multiple Excel sheets using Python, presenting two practical solutions—one with openpyxl and another with pandas—complete with code snippets, explanations, and a final tip for sorting results by score.
This article walks through a real‑world Pandas question from a Python community, presents two solution approaches—including a double‑for‑loop implementation—provides the full code snippet, and explains how the fix resolves the data lookup problem.
JavaScript Generators, introduced in ES6, allow functions to pause and resume execution, yielding multiple values; this article explains their syntax, basic usage, advanced features like yield* and data exchange, and demonstrates practical scenarios such as asynchronous flow control, memory-efficient data processing, and state machine implementation.
This guide demonstrates how to use Python's pandas library to merge multiple Excel workbooks—whether from a single folder, across several sheets, or into a single DataFrame—and then convert the resulting data into JSON format, including detailed code examples and explanations.
This article walks through a real‑world Pandas data‑processing problem, presenting two concise Python solutions—using the built‑in pow function and a list comprehension—complete with code snippets and practical tips for handling similar issues.
This article introduces QueryList, a modern PHP content‑scraping tool that uses CSS selectors instead of regex, explains its two versions (V3 and V4), shows how to install it via Composer, demonstrates basic crawling code and various collection methods such as flatten, take, reverse, filter, map, and multi‑request concurrency.
This article walks through a real‑world Pandas grouping issue, presents multiple community‑sourced strategies, and provides a concise code solution that demonstrates how to generate sequential identifiers and concatenate columns for accurate voucher numbering.
This guide demonstrates how to use Pandas to read an Excel file, locate rows by name using index and loc, and resolve the common 'numexpr' version warning by upgrading the package, providing clear code examples and step‑by‑step instructions.
The article explains how excessive intermediate tables generated by reporting workloads degrade database storage and performance, and proposes using the SPL data‑processing tool to store these intermediate results as external files, thereby reducing capacity pressure, improving I/O speed, and simplifying management.
This article demonstrates how to replace typical Excel operations such as VLOOKUP, pivot tables, duplicate removal, missing‑value handling, multi‑condition filtering, fuzzy matching, column splitting, outlier replacement, grouping and labeling with concise Python pandas code to streamline data analysis workflows.
The article explains how Blink, Alibaba’s optimized Flink variant, uses dynamic tables and streaming‑SQL to ingest mobile telemetry via source tables, compute per‑minute metrics such as API success rates with tumbling windows, and write results to Alibaba Cloud Log Service, enabling real‑time dashboards and extensible use cases like fraud detection.
This tutorial introduces PhpSpreadsheet, a pure‑PHP library for handling Excel files, covering Composer installation, namespace imports, reading and writing worksheets, applying styles, and various cell operations with complete code examples for building and formatting spreadsheets.
This article outlines a student project to create a Python-powered electronic attendance system, detailing required CSV data formats, core functions such as loading data, login, record writing, and querying, along with task requirements, additional features, and complete code examples.
Learn how to automate bulk find‑replace operations in Excel using Python’s pandas library, with a step‑by‑step walkthrough, complete code example, handling of NaN/NaT values, and the resulting output that streamlines repetitive data‑processing tasks.
This tutorial explains how to use Python's xlrd, xlwt, and openpyxl libraries to read, write, and append data in both XLS and XLSX Excel formats, providing complete code examples, performance considerations, and test results for each operation.
This article walks through a pandas data‑processing problem, showing how to concatenate province, city, and district fields using vectorized string operations with clear code examples and resulting output to help readers apply the technique.
This article walks through a real‑world Pandas problem posted in a Python community, showing how to read multiple Excel sheets, reshape the data, rename columns, and filter the result to match a desired table layout, complete with code and screenshots.
This article provides a comprehensive guide to using the openpyxl library for creating, opening, editing, and saving Excel workbooks in Python, covering worksheet management, cell operations, merging, iteration, and efficient handling of large files with code examples.
This article describes how the Tongtian Tower platform breaks traditional R&D barriers by enabling zero‑code, one‑click conversion of design drafts into production‑ready floor layouts, detailing the underlying architecture, data processing pipeline, core capabilities, challenges faced, and future optimization plans.
This guide explains how unemployed developers can use Python, especially web‑scraping techniques, to secure freelance gigs by leveraging various platforms, community groups, and effective order‑taking strategies while warning about common pitfalls and anti‑scraping challenges.
This article introduces Java 8's Stream API, explains its SQL‑like pipeline operations such as filter, map, sorted, collect and parallelStream, and demonstrates practical PO/VO processing with comprehensive code examples and performance considerations.
This article walks through a real‑world Pandas problem where a column of tag strings is parsed, extracted, and transformed into dummy variables with get_dummies, providing clear code snippets and explanations to help Python developers solve similar data‑processing challenges.
This article walks through a real‑world pandas question, shows the original JSON data, presents a concise Python script to normalize and explode the data, highlights common pitfalls, and demonstrates the successful output with clear screenshots.
This article walks through transforming a list of Python dictionaries into a consolidated structure using basic loops, itertools grouping, and Pandas, providing clear code examples and visual results to help readers solve common dictionary‑handling challenges.
This article walks through a Python community member's JSON‑processing problem, explores several solutions using pandas, jsonpath, and regular expressions, and presents a final script that normalizes the data into a tidy CSV for further analysis.
This article details the design and implementation of a real‑time data warehouse for an advertising platform, covering business background, challenges, a Lambda‑based architecture, Flink stream processing setup, ETL logic, sink handling, and performance results, concluding with future improvement directions.
This article walks through a Python tutorial that generates a random list, sorts it, and calculates the minimum, maximum, sum, and average values, providing clear explanations, code snippets, and output screenshots for readers to follow along.
This article demonstrates how to replace tedious manual Excel reporting by using Python libraries pandas and xlwings to read multiple sheets, merge data, write the combined DataFrame back to Excel, and apply conditional formatting such as font colors, borders, and cell shading based on statistical thresholds.
This article provides a comprehensive overview of the big data ecosystem, detailing nine core technology categories—from data collection and storage to computation, analysis, scheduling, and underlying infrastructure—along with tool comparisons, selection guidelines to help readers quickly build a complete big data knowledge system.
The talk traces the evolution of API Gateway architectures and the engineering practices—design patterns, deployment strategies, and operational considerations—required for scalable, reliable services, then demonstrates how ClickHouse can be leveraged for massive data workloads, highlighting practical scenarios, performance optimizations, and key lessons learned.
This article walks through a common Pandas time‑handling issue, showing how to convert a timestamp column to a YYYY‑MM‑DD date format with concise code, complete with example data, output screenshots, and a brief conclusion for Python learners.
This article explains how to compute indirect shareholding ratios between companies by generating synthetic relationship data, cleaning and normalizing it with multiprocessing, constructing a weighted directed graph using NetworkX, and applying a matrix‑based algorithm to derive the final ownership matrix.
This article walks through a Pandas date‑handling challenge, presenting two robust methods using MonthEnd to convert month strings, discussing edge cases where month‑end dates shift, and sharing community‑derived optimizations and pure‑Python alternatives to help Python developers process dates accurately.
This article outlines six typical errors beginners make with Pandas—slow CSV reads, lack of vectorization, improper dtypes, ignoring styling, inefficient CSV saving, and not consulting documentation—and provides faster alternatives, memory‑saving techniques, and best‑practice tips for handling large datasets.
This article explains the concept and challenges of data lakes, describes the “impossible triangle” of storage, compute, and cost, and introduces the open‑source SPL engine that provides multi‑source, file‑based, high‑performance computing to overcome those limitations.
This article introduces nine popular tools—including JSONLint, JSONCompare, JTC, ijson, and others—that help developers validate, format, compress, compare, and edit JSON data, providing both online services and IDE plugins for more efficient JSON handling.
This article provides a detailed tutorial on NumPy array manipulation in Python, covering iteration with np.nditer, reshaping, flattening, transposition, axis swapping, broadcasting, stacking, concatenation, splitting, resizing, appending, inserting, deleting, unique element extraction, string utilities, arithmetic, statistical analysis, sorting, searching, and file I/O, each illustrated with concise code examples.
This article walks through a fan's data‑processing question by presenting two practical Python approaches—renaming columns then summing, and using NumPy addition—to combine matching columns from separate DataFrames, complete with visual code examples.
This article presents a Python utility that reads an original class schedule Excel file, processes the data with pandas, and generates separate class‑wise and teacher‑wise timetables saved as Excel files, all wrapped in a simple PySimpleGUI interface requiring a login.
This article walks through two practical methods for converting a Python list into the first column of an Excel file—first using openpyxl for direct cell writing, then leveraging pandas for a more concise DataFrame approach, plus additional pandas tricks for column manipulation.
This article examines the intricate data processing challenges of a mobile message client, outlines seven key complexities, proposes a unified abstraction layer to reduce code by 60%, and details a modular solution—including MergeDispatcher, Calculator, and DataStructure—that separates computation from data fetching for improved consistency and scalability.
This guide explains how to install jq on CentOS 8, introduces its command‑line syntax, and provides step‑by‑step examples for organizing JSON output, accessing object properties, and iterating over arrays using jq filters.
This guide walks you through installing openpyxl, opening and creating workbooks, accessing sheets and cells, iterating over ranges, modifying data, applying styles, and saving Excel files using Python, complete with code examples and visual illustrations.
This article details how Baidu's search middle platform migrated from script‑based processing to a serverless business‑framework architecture, outlining the technical challenges, design of data ingestion, processing, scheduling, and control layers, and summarizing the efficiency, cost, and performance gains achieved.
This tutorial walks through Java 8 Stream fundamentals, covering creation, intermediate operations (filter, map, flatMap, reduce), terminal actions, collectors, grouping, sorting, and parallel streams, illustrated with 20 practical code examples that transform and analyze collections of objects such as employee records.
This guide explains Sentry SDK's data‑processing context, detailing the default‑disabled send‑default‑pii option, the before‑send hook for custom redaction, and comprehensive size limits for event fields, metadata, and attachments, with links to official documentation.
This article presents a thorough collection of frequently used pandas commands for importing data, preprocessing, cleaning, transforming, analyzing, and visualizing datasets, complete with code examples and explanations to help Python developers perform efficient data analysis tasks.
This article demonstrates how to use Python and pandas to automatically add a column containing each workbook or sheet name to every sheet in multiple Excel files and then merge all the data into a single file, dramatically reducing manual effort.
This guide demonstrates how to efficiently read, write, and convert CSV, JSON, and XML data using Python's built‑in modules, pandas, and third‑party libraries, providing clear code examples for each format and showcasing best practices for data processing.
This article introduces Spark's core concepts, explains the RDD abstraction and its four main properties, and details the roles of DAGScheduler, SchedulerBackend, TaskScheduler, and ExecutorBackend, providing practical insights for beginners and intermediate users in big‑data processing.
This guide explains how to use Alibaba's EasyExcel library for efficient, low‑memory Excel import and export in Java, covering its core features, common annotations, Maven dependencies, listener implementation, and practical code examples for both HTTP‑based and local file operations.
This article introduces SPL, an open‑source Java‑embeddable computation library that outperforms traditional embedded databases and DataFrame tools by handling both tabular and nested JSON data, supporting JDBC, SQL‑like queries, multi‑source integration, and persistent .btx files with concise code examples.
This tutorial demonstrates how to use the Python PyWebIO library to create a web interface for uploading, cleaning, and querying pandas data—adding buttons, displaying tables, and handling continuous input—without writing any front-end HTML or JavaScript code.
This article walks you through the complete process of automating Excel report generation with Python, covering Excel fundamentals, a five‑step workflow, practical code examples using pandas and openpyxl, and techniques for merging results into a single workbook or multiple sheets.
The article describes the background, challenges, architecture, operator design, DAG processing, tag persistence, and business applications of a real-time dynamic tag processing platform (CDP) built to improve revenue growth and cost reduction for Trip.com's international operations.
This tutorial walks through Java 8 Stream and Lambda features, explaining stream concepts, intermediate and terminal operations, and demonstrating twenty practical examples—including creation, filtering, mapping, reduction, collection, sorting, and combining—using an employee class to illustrate each operation.
This article provides an in-depth exploration of JavaScript's Streams API, covering core concepts like readable/writable/transform streams, backpressure, and practical applications such as video processing and data compression.
This article provides an in‑depth tutorial on Java 8 Stream API, covering its concepts, creation methods, common operations such as filtering, mapping, reducing, collecting, sorting, and grouping, along with numerous runnable code examples that demonstrate how to process collections efficiently using streams.
This article provides a comprehensive overview of Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, chunk processing, skip strategies, and practical guidelines for building robust batch processing solutions in Java.
The article explores how low‑code platforms can complement traditional algorithm development, enhance collaboration between business users and engineers, and accelerate big‑data and AI initiatives by improving data cleaning, modular design, and feedback loops, while highlighting the trade‑offs of abstraction and flexibility.
This tutorial explains how to package reusable Python functions into importable modules and provides a step‑by‑step guide, including code, to merge multiple Excel files into a single workbook using pandas, enabling easier reuse and automation of common data‑processing tasks.
The article explains why Java 8 Stream API is still valuable despite SQL offering similar functions, detailing Stream creation, terminal and intermediate operations, collectors, matching, and practical scenarios where Streams simplify data processing across microservices.
This article introduces five lesser‑known but powerful NumPy functions—reshape with -1, argpartition, clip, extract, and setdiff1d—explaining their behavior, showcasing code examples, and highlighting how they simplify complex data manipulation tasks.
This tutorial explores jq's advanced capabilities, demonstrating how to combine filters with the pipe operator, use functions like keys, length, select, map, and join, and transform JSON data into new structures and formats through practical command‑line examples.
