BestHub
Sign in
Tagged articles

data cleaning

139 articles · Page 2 of 2
Python Crawling & Data Mining
Python Crawling & Data Mining
Aug 23, 2021 · Fundamentals

How to Clean and Analyze Messy Taobao Data with Python Regex and Pandas

This article walks through cleaning chaotic Taobao CSV data using Python's regular expressions and pandas, removing unwanted characters with stop‑words, performing word segmentation, and generating word‑frequency statistics through both a classic approach and a pandas‑optimized method, complete with code snippets and visual results.

RegexWord Frequencydata cleaning
0 likes · 10 min read
How to Clean and Analyze Messy Taobao Data with Python Regex and Pandas
Python Programming Learning Circle
Python Programming Learning Circle
Aug 6, 2021 · Fundamentals

A Comprehensive List of Commonly Used Pandas Functions Categorized by Purpose

This article presents a curated collection of 100 frequently used pandas functions, organized into six categories—statistical aggregation, data cleaning, data selection, plotting and element‑wise operations, time‑series utilities, and miscellaneous helpers—providing concise Chinese explanations for each function’s purpose.

Data manipulationPandasPython
0 likes · 10 min read
A Comprehensive List of Commonly Used Pandas Functions Categorized by Purpose
MaGe Linux Operations
MaGe Linux Operations
Jul 29, 2021 · Databases

How to Efficiently Remove Duplicate Rows in Large MySQL Tables

This article explains why a naïve Python script for deduplicating millions of rows is too slow, then walks through a series of MySQL queries—including how to identify duplicate names, avoid the 1093 error, and delete duplicates while keeping a single representative row—demonstrating fast, reliable cleanup of large tables.

SQLdata cleaningdatabase optimization
0 likes · 5 min read
How to Efficiently Remove Duplicate Rows in Large MySQL Tables
Architect's Tech Stack
Architect's Tech Stack
Jul 26, 2021 · Databases

Efficient Methods to Remove Duplicate Rows in MySQL Tables

This article explains how to identify and delete duplicate records in large MySQL tables, discusses why a naïve delete fails, and provides two robust SQL solutions—one that removes all duplicates and another that retains a single row per duplicated key—demonstrating fast execution even on tables with hundreds of thousands of rows.

SQLdata cleaningduplicate removal
0 likes · 5 min read
Efficient Methods to Remove Duplicate Rows in MySQL Tables
Architects' Tech Alliance
Architects' Tech Alliance
Mar 1, 2021 · Cloud Computing

Practices for Data Cleaning and Cutover Consistency in Cross‑Cloud Migration

This article explains the technical details of data cleaning, dirty‑data handling, and three methods—database read‑only, application termination, and network ACL isolation—to ensure data consistency during the data‑regulation and cutover phases of cross‑cloud migration, illustrated with real‑world case studies.

ACL isolationData Consistencycloud migration
0 likes · 12 min read
Practices for Data Cleaning and Cutover Consistency in Cross‑Cloud Migration
NetEase Smart Enterprise Tech+
NetEase Smart Enterprise Tech+
Jan 14, 2021 · Big Data

How Yidun Achieves Real-Time, High-Performance Public-Opinion Data Cleaning with Groovy and JVM

Yidun’s public-opinion monitoring platform transforms massive raw web data into a unified format by separating dynamic Groovy-script-driven cleaning from static processing, achieving real-time source integration, high throughput, scalability, and high availability while addressing format diversity, team coordination, and performance-flexibility trade-offs.

ETLGroovyJVM
0 likes · 5 min read
How Yidun Achieves Real-Time, High-Performance Public-Opinion Data Cleaning with Groovy and JVM
Top Architect
Top Architect
Jan 13, 2021 · Big Data

Migrating Over 2 Billion MySQL Records to the Cloud with Kafka and BigQuery

Facing a MySQL table with over 2 billion continuously growing rows, the team designed a cloud‑based solution using Kafka to stream data into Google BigQuery, applied partitioned schemas and data cleaning to improve query performance, reduce storage costs, and avoid downtime.

BigQueryData MigrationPartitioning
0 likes · 8 min read
Migrating Over 2 Billion MySQL Records to the Cloud with Kafka and BigQuery
Python Programming Learning Circle
Python Programming Learning Circle
Dec 18, 2020 · Fundamentals

Data Exploration and Cleaning: Core Concepts, Steps, and Example Workflow

This article explains the purpose of data exploration and cleaning, outlines core analysis tasks, details missing‑value and outlier handling techniques—including various imputation methods—and illustrates the complete workflow with example images and a histogram‑based distribution analysis.

data cleaningdata explorationdata preprocessing
0 likes · 3 min read
Data Exploration and Cleaning: Core Concepts, Steps, and Example Workflow
Python Crawling & Data Mining
Python Crawling & Data Mining
Dec 2, 2020 · Artificial Intelligence

Scrape, Clean, and Visualize Tencent Video Comments with Python – A Full Guide

This article walks through using Python to crawl Tencent Video's "Offer" season 2 comments, merge and clean the CSV data, perform exploratory analysis, generate visualizations and word clouds, and apply Baidu's open‑source NLP model for sentiment scoring, providing complete code snippets for each step.

PythonSentiment AnalysisWeb Scraping
0 likes · 16 min read
Scrape, Clean, and Visualize Tencent Video Comments with Python – A Full Guide
Python Programming Learning Circle
Python Programming Learning Circle
Nov 10, 2020 · Backend Development

Detailed Job Description Extraction and Data Cleaning with Python and MongoDB

This article explains how to scrape detailed job description and address information from online job portals, use Python libraries such as requests, BeautifulSoup4, and pymongo for crawling, and then clean and normalize the collected data including publish dates, salaries, and work‑experience levels before storing it in MongoDB.

BeautifulSoupMongoDBPython
0 likes · 8 min read
Detailed Job Description Extraction and Data Cleaning with Python and MongoDB
Liangxu Linux
Liangxu Linux
Oct 20, 2020 · Operations

Mastering AWK: Quick Commands to Clean, Filter, and Analyze Text Files

This guide demonstrates how to use AWK’s sub, gsub, and pattern‑matching features to delete blank lines, trim whitespace, reverse file order, add line numbers, count or filter specific strings, replace text, and combine AWK with other Linux utilities for everyday text‑processing tasks.

Shell scriptingawkdata cleaning
0 likes · 7 min read
Mastering AWK: Quick Commands to Clean, Filter, and Analyze Text Files
Efficient Ops
Efficient Ops
Jul 13, 2020 · Operations

What 13,966 Ops Job Listings Reveal About Salary, Skills, and Hot Cities

This article analyzes 13,966 Chinese operations‑engineer job postings scraped from 51job, cleaning the data with Python and Pandas, then visualizing industry demand, city concentration, salary ranges, education requirements, company size distribution, and keyword trends to guide job seekers and recruiters.

Data VisualizationPandasPython
0 likes · 14 min read
What 13,966 Ops Job Listings Reveal About Salary, Skills, and Hot Cities
DataFunTalk
DataFunTalk
Jul 3, 2020 · Artificial Intelligence

Confident Learning: Detecting and Cleaning Noisy Labels with cleanlab

This article introduces confident learning, a principled framework for identifying and correcting mislabeled data in machine‑learning datasets, explains its three‑step process (count, clean, re‑training), demonstrates usage of the open‑source cleanlab library with code examples, and presents experimental results showing its effectiveness on benchmarks such as CIFAR‑10 and ImageNet.

cleanlabconfident learningdata cleaning
0 likes · 13 min read
Confident Learning: Detecting and Cleaning Noisy Labels with cleanlab
Laravel Tech Community
Laravel Tech Community
Jun 5, 2020 · Backend Development

Using PHP trim() to Remove Whitespace and Specified Characters

This article explains PHP's trim() function, its parameters and default character list, and provides multiple code examples showing how to trim strings, binary data, and array elements, including custom functions and array_walk usage, with the resulting outputs displayed.

data cleaningphp-functionsstring-manipulation
0 likes · 3 min read
Using PHP trim() to Remove Whitespace and Specified Characters
Python Crawling & Data Mining
Python Crawling & Data Mining
Apr 7, 2020 · Fundamentals

Master 50 Essential Pandas Exercises to Boost Your Data Skills

This article presents a comprehensive collection of 50 pandas practice problems that guide you through creating Series and DataFrames, performing basic and advanced indexing, grouping, aggregation, data cleaning, hierarchical indexing, and visualisation, each illustrated with clear Python code examples.

DataFramedata cleaningseries
0 likes · 19 min read
Master 50 Essential Pandas Exercises to Boost Your Data Skills
Aikesheng Open Source Community
Aikesheng Open Source Community
Feb 10, 2020 · Databases

Handling Duplicate Data in MySQL: Techniques and Examples

This article explains how to identify and remove various kinds of duplicate data in MySQL—including fully duplicated rows, records with duplicate non‑key columns, and unwanted whitespace inside fields—by using SQL statements, table cloning, OS utilities, and regular‑expression updates, with performance measurements for each method.

Database MigrationSQLdata cleaning
0 likes · 13 min read
Handling Duplicate Data in MySQL: Techniques and Examples
Python Crawling & Data Mining
Python Crawling & Data Mining
Oct 9, 2019 · Fundamentals

Master Pandas Basics: From DataFrames to Quick Data Insights

This tutorial introduces Pandas fundamentals, covering installation, DataFrame creation, reading and storing CSV/Excel files, quick data inspection, column manipulation, handling different data types, and basic time series operations, providing a concise roadmap for beginners to start data analysis with Python.

DataFramedata cleaning
0 likes · 13 min read
Master Pandas Basics: From DataFrames to Quick Data Insights
Alibaba Cloud Developer
Alibaba Cloud Developer
Sep 12, 2019 · Artificial Intelligence

How a Simple Learning‑Rate Trick Detects 90% of Noisy Labels in Image Data

Training deep neural networks on large‑scale weakly labeled image data suffers from noisy annotations that degrade performance, but a simple algorithm that adjusts the learning‑rate during training can automatically identify up to 90% of noisy samples, improving dataset cleanliness and model accuracy without manual intervention.

Deep Learningdata cleaningimage classification
0 likes · 15 min read
How a Simple Learning‑Rate Trick Detects 90% of Noisy Labels in Image Data
Tencent Advertising Technology
Tencent Advertising Technology
May 8, 2019 · Artificial Intelligence

Experience Sharing of the 2019 Tencent Advertising Algorithm Competition – Week 1 Champion’s Insights

The week‑1 champion of the 2019 Tencent Advertising Algorithm Competition shares practical experience on data cleaning, feature engineering, model selection (including LightGBM and deep learning), validation strategies, and tips for handling massive ad exposure logs to achieve high SMAPE and monotonicity scores.

AdvertisingAlgorithm CompetitionLightGBM
0 likes · 6 min read
Experience Sharing of the 2019 Tencent Advertising Algorithm Competition – Week 1 Champion’s Insights
DataFunTalk
DataFunTalk
Aug 21, 2018 · Artificial Intelligence

iQIYI Traffic Anti-Cheat: Techniques, System Architecture, and Future Directions

This article provides a comprehensive overview of iQIYI's traffic anti‑cheat mechanisms, covering definitions of fraudulent traffic, industry challenges, data cleaning relationships, system design, rule‑based and machine‑learning solutions, feature engineering, model evaluation, monitoring, service applications, and future prospects.

Traffic analysisanti-fraudbig data
0 likes · 11 min read
iQIYI Traffic Anti-Cheat: Techniques, System Architecture, and Future Directions
Qunar Tech Salon
Qunar Tech Salon
Jul 27, 2018 · Information Security

Design and Features of an Anti‑Crawling Platform for Large‑Scale Services

The article describes the goals, architecture, core functions, and key characteristics of a comprehensive anti‑crawling platform that systematizes strategy management, data cleaning, monitoring, and rapid response to protect APIs and improve data reliability for large‑scale online services.

PlatformSecurityanti‑crawling
0 likes · 10 min read
Design and Features of an Anti‑Crawling Platform for Large‑Scale Services
MaGe Linux Operations
MaGe Linux Operations
Apr 26, 2018 · Databases

Find Duplicate Rows in MySQL: Simple Queries for Beginners

This article shows how to identify and remove duplicate rows in a MySQL table by defining duplication, using GROUP BY with HAVING, creating temporary tables with MIN, and applying various techniques—including UNION, nested subqueries, and joins—to handle single‑column and multi‑column duplicate detection.

GROUP BYHAVINGSQL
0 likes · 11 min read
Find Duplicate Rows in MySQL: Simple Queries for Beginners
21CTO
21CTO
Nov 6, 2017 · Artificial Intelligence

Predict Retail Sales Without Coding: A Complete KNIME Tutorial

This step‑by‑step guide shows beginners how to use the GUI‑driven KNIME platform to import, clean, visualize, and model the BigMart sales dataset, enabling accurate retail sales predictions without writing any code.

KNIMEdata cleaningno-code
0 likes · 12 min read
Predict Retail Sales Without Coding: A Complete KNIME Tutorial
21CTO
21CTO
Oct 14, 2017 · Backend Development

How etlpy Simplifies Python Web Scraping and Data Cleaning in Under 500 Lines

etlpy is a lightweight Python framework that lets you define web‑crawling and data‑cleaning pipelines via XML, using generators for streaming, built‑in thread pools for parallelism, and a plug‑in architecture that handles everything from regex parsing to JSON conversion, all within a single 500‑line core file.

ETLWeb Scrapingdata cleaning
0 likes · 14 min read
How etlpy Simplifies Python Web Scraping and Data Cleaning in Under 500 Lines
MaGe Linux Operations
MaGe Linux Operations
Aug 18, 2017 · Backend Development

How to Scrape Douban Movie Reviews and Visualize Them with a Word Cloud in Python

This tutorial walks through using Python 3.5 to fetch the latest movies from Douban, extract their IDs and titles, crawl user comments, clean the text with regular expressions, segment Chinese words using Jieba, remove stopwords, compute word frequencies, and finally generate a word‑cloud visualization of the reviews.

data cleaningdoubanweb-scraping
0 likes · 13 min read
How to Scrape Douban Movie Reviews and Visualize Them with a Word Cloud in Python
Aotu Lab
Aotu Lab
Dec 6, 2016 · Frontend Development

How XCEL Turns Excel Data Cleaning into a High‑Performance Cross‑Platform App

XCEL is a cross‑platform Excel data‑cleaning tool built with Electron and Vue that visualizes filtering, leverages multi‑process architecture, optimizes rendering and memory usage, and provides detailed implementation steps, performance tricks, and code examples for developers.

Exceldata cleaningperformance optimization
0 likes · 25 min read
How XCEL Turns Excel Data Cleaning into a High‑Performance Cross‑Platform App
Architects' Tech Alliance
Architects' Tech Alliance
Dec 3, 2016 · Fundamentals

Effective Data Cleaning Practices and Tips

This article provides practical guidance on data cleaning, covering the importance of data wrangling, using assertions, handling incomplete records, checkpointing, testing on subsets, logging, optional raw data storage, and validating the cleaned dataset to ensure reliable downstream analysis.

CheckpointValidationassertions
0 likes · 7 min read
Effective Data Cleaning Practices and Tips