Facing a MySQL table with over 2 billion continuously growing rows, the team designed a cloud‑based solution using Kafka to stream data into Google BigQuery, applied partitioned schemas and data cleaning to improve query performance, reduce storage costs, and avoid downtime.
This article explains the purpose of data exploration and cleaning, outlines core analysis tasks, details missing‑value and outlier handling techniques—including various imputation methods—and illustrates the complete workflow with example images and a histogram‑based distribution analysis.
This article walks through using Python to crawl Tencent Video's "Offer" season 2 comments, merge and clean the CSV data, perform exploratory analysis, generate visualizations and word clouds, and apply Baidu's open‑source NLP model for sentiment scoring, providing complete code snippets for each step.
This article introduces the FuzzyWuzzy Python library, explains its core functions and the Levenshtein‑based similarity metrics, and provides step‑by‑step code examples for fuzzy matching of company and province names, including a reusable DataFrame merge function.
This article explains how to scrape detailed job description and address information from online job portals, use Python libraries such as requests, BeautifulSoup4, and pymongo for crawling, and then clean and normalize the collected data including publish dates, salaries, and work‑experience levels before storing it in MongoDB.
This guide demonstrates how to use AWK’s sub, gsub, and pattern‑matching features to delete blank lines, trim whitespace, reverse file order, add line numbers, count or filter specific strings, replace text, and combine AWK with other Linux utilities for everyday text‑processing tasks.
This article analyzes 13,966 Chinese operations‑engineer job postings scraped from 51job, cleaning the data with Python and Pandas, then visualizing industry demand, city concentration, salary ranges, education requirements, company size distribution, and keyword trends to guide job seekers and recruiters.
This article introduces confident learning, a principled framework for identifying and correcting mislabeled data in machine‑learning datasets, explains its three‑step process (count, clean, re‑training), demonstrates usage of the open‑source cleanlab library with code examples, and presents experimental results showing its effectiveness on benchmarks such as CIFAR‑10 and ImageNet.
This article explains PHP's trim() function, its parameters and default character list, and provides multiple code examples showing how to trim strings, binary data, and array elements, including custom functions and array_walk usage, with the resulting outputs displayed.
This article introduces the cpca Python module that parses free‑form Chinese address strings into province, city, and district fields, explains how to install it, demonstrates basic and advanced usage with code examples, and describes optional full‑text and map‑drawing features.
This tutorial demonstrates how to download, read, explore, visualize, and preprocess the HR attrition dataset with pandas, covering tasks such as duplicate removal, missing‑value handling, categorical encoding, normalization, and conditional column updates to prepare the data for machine‑learning modeling.
This article presents a comprehensive collection of 50 pandas practice problems that guide you through creating Series and DataFrames, performing basic and advanced indexing, grouping, aggregation, data cleaning, hierarchical indexing, and visualisation, each illustrated with clear Python code examples.
This article explains how to identify and remove various kinds of duplicate data in MySQL—including fully duplicated rows, records with duplicate non‑key columns, and unwanted whitespace inside fields—by using SQL statements, table cloning, OS utilities, and regular‑expression updates, with performance measurements for each method.
This article introduces essential Pandas techniques for data import, cleaning, type conversion, and merging, providing clear code examples that demonstrate reading from MySQL, handling missing values, transforming columns, and combining multiple DataFrames for comprehensive data analysis.
This tutorial introduces Pandas fundamentals, covering installation, DataFrame and Series concepts, creating, reading, and storing data, quick inspection methods, essential column operations, and handling of common data types such as strings, numerics, and dates.
This tutorial introduces Pandas fundamentals, covering installation, DataFrame creation, reading and storing CSV/Excel files, quick data inspection, column manipulation, handling different data types, and basic time series operations, providing a concise roadmap for beginners to start data analysis with Python.
Training deep neural networks on large‑scale weakly labeled image data suffers from noisy annotations that degrade performance, but a simple algorithm that adjusts the learning‑rate during training can automatically identify up to 90% of noisy samples, improving dataset cleanliness and model accuracy without manual intervention.
The week‑1 champion of the 2019 Tencent Advertising Algorithm Competition shares practical experience on data cleaning, feature engineering, model selection (including LightGBM and deep learning), validation strategies, and tips for handling massive ad exposure logs to achieve high SMAPE and monotonicity scores.
Developers can boost efficiency by using Excel to concatenate column values into SQL IN clauses or C# List syntax, locate duplicate entries, split cell contents by delimiters, and permanently disable automatic hyperlink creation, with step‑by‑step visual guides illustrating each technique.
This article provides a comprehensive overview of iQIYI's traffic anti‑cheat mechanisms, covering definitions of fraudulent traffic, industry challenges, data cleaning relationships, system design, rule‑based and machine‑learning solutions, feature engineering, model evaluation, monitoring, service applications, and future prospects.
The article describes the goals, architecture, core functions, and key characteristics of a comprehensive anti‑crawling platform that systematizes strategy management, data cleaning, monitoring, and rapid response to protect APIs and improve data reliability for large‑scale online services.
This article shows how to identify and remove duplicate rows in a MySQL table by defining duplication, using GROUP BY with HAVING, creating temporary tables with MIN, and applying various techniques—including UNION, nested subqueries, and joins—to handle single‑column and multi‑column duplicate detection.
This step‑by‑step guide shows beginners how to use the GUI‑driven KNIME platform to import, clean, visualize, and model the BigMart sales dataset, enabling accurate retail sales predictions without writing any code.
etlpy is a lightweight Python framework that lets you define web‑crawling and data‑cleaning pipelines via XML, using generators for streaming, built‑in thread pools for parallelism, and a plug‑in architecture that handles everything from regex parsing to JSON conversion, all within a single 500‑line core file.
This tutorial walks through using Python 3.5 to fetch the latest movies from Douban, extract their IDs and titles, crawl user comments, clean the text with regular expressions, segment Chinese words using Jieba, remove stopwords, compute word frequencies, and finally generate a word‑cloud visualization of the reviews.
The article presents Baidu's end‑to‑end third‑party sentiment feedback solution, detailing its evolution from waterfall to agile and lean development, and describing the three‑stage pipeline of data acquisition, cleansing, and warehousing that enables real‑time product quality loops.
XCEL is a cross‑platform Excel data‑cleaning tool built with Electron and Vue that visualizes filtering, leverages multi‑process architecture, optimizes rendering and memory usage, and provides detailed implementation steps, performance tricks, and code examples for developers.
This article provides practical guidance on data cleaning, covering the importance of data wrangling, using assertions, handling incomplete records, checkpointing, testing on subsets, logging, optional raw data storage, and validating the cleaned dataset to ensure reliable downstream analysis.
The article reveals how algorithm engineers spend most of their time on data cleaning, manual feature engineering, and business integration, exposing a gap between the glamorous perception of the role and the gritty, data‑centric work that actually defines it.
