Skip to content
Tagged articles

data cleaning

147 articles · Page 1 of 2
Data Integration and Governance
Data Integration and Governance
Oct 13, 2026 · Big Data

What Makes a Dataset 'Good' for AI? Six Quality Dimensions & the Engineering Pipeline

The article defines high-quality datasets in the AI era by six dimensions—accuracy, completeness, consistency, timeliness, understandability, and traceability—explains why AI amplifies data quality requirements, and outlines a five-step engineering pipeline (aggregation, cleaning, unification, validation, traceability) to transform raw data into reliable AI-ready datasets, emphasizing that continuous data engineering, not data hoarding, enables AI success.

AI data preparationData EngineeringFineDataLink
0 likes · 14 min read
What Makes a Dataset 'Good' for AI? Six Quality Dimensions & the Engineering Pipeline
BanTech Think Tank
BanTech Think Tank
Sep 15, 2026 · Backend Development

From 4 Hours to 40 Minutes: Postal Savings Bank's Cross-Unit Data Cleaning Breakthrough

Postal Savings Bank developed a parameter-driven, group-concurrent cross-unit data cleaning framework for its distributed core remittance system, reducing cleaning time for hundreds of millions of records from 4 hours to 40 minutes while ensuring consistency and minimizing business impact.

Distributed Systemsconcurrency-controlcore banking
0 likes · 9 min read
From 4 Hours to 40 Minutes: Postal Savings Bank's Cross-Unit Data Cleaning Breakthrough
Subtle Storm
Subtle Storm
Sep 9, 2026 · Artificial Intelligence

Zero-Code Data Analysis: My 5-Minute WorkBuddy Experiment Reveals Hidden Insights

A programmer pretends to be a non-coder and uses natural language to instruct WorkBuddy to clean, analyze, and visualize WeChat article data, discovering readers share personal stories more than technical content, proving that asking the right questions matters more than coding skills.

AI-assisted analyticsWorkBuddydata cleaning
0 likes · 7 min read
Zero-Code Data Analysis: My 5-Minute WorkBuddy Experiment Reveals Hidden Insights
Data Integration and Governance
Data Integration and Governance
Jul 16, 2026 · Fundamentals

Data Cleaning Demystified: 10 Essential Techniques from Missing Values to Outlier Handling

The article walks through a complete data‑cleaning workflow—starting with why rushed analysis fails, then detailing ten practical methods for handling missing, duplicate, malformed, and outlier data, standardizing formats and units, validating logic and relationships, and finally cementing reusable rules so enterprise data stays trustworthy for reporting and analytics.

ETLdata cleaningduplicate removal
0 likes · 17 min read
Data Cleaning Demystified: 10 Essential Techniques from Missing Values to Outlier Handling
Machine Heart
Machine Heart
Jul 9, 2026 · Artificial Intelligence

How DM0.5 Brings Zero‑Shot, Long‑Term Memory, and Robustness to Real‑World VLA

DM0.5 advances the VLA paradigm by adding zero‑shot capability, efficient fine‑tuning, up to 60‑second memory, stronger resistance to visual and human interference, and cross‑robot transfer, achieved through long‑history modeling, embodied reasoning tasks, trajectory‑alignment supervision, and rigorous multi‑source data cleaning pipelines.

Embodied AILong-term MemoryRobotics
0 likes · 13 min read
How DM0.5 Brings Zero‑Shot, Long‑Term Memory, and Robustness to Real‑World VLA
Data Integration and Governance
Data Integration and Governance
Jul 6, 2026 · Fundamentals

Why Messy Data Demands Immediate Cleaning: A Complete Data‑Cleaning Workflow

Many analysts rush to build dashboards on raw data, only to discover mismatched numbers and business push‑back, because the data was never properly cleaned; this article outlines a structured, rule‑based, verifiable and reusable data‑cleaning process that starts with clear goals, proceeds through inventory, rule definition, standardization, mapping, validation, and ends with documented deliverables.

AnalyticsETLdata cleaning
0 likes · 13 min read
Why Messy Data Demands Immediate Cleaning: A Complete Data‑Cleaning Workflow
Data Integration and Governance
Data Integration and Governance
Jun 30, 2026 · Fundamentals

9 Essential Data Cleaning Techniques Every Analyst Must Master

Before visualizing or modeling, analysts must first resolve common data quality issues—duplicate records, inconsistent formats, missing or abnormal values, and mismatched definitions—by applying nine systematic cleaning steps that turn raw, chaotic data into reliable, comparable, and reusable information.

Data Engineeringdata cleaningdata validation
0 likes · 20 min read
9 Essential Data Cleaning Techniques Every Analyst Must Master
Data Integration and Governance
Data Integration and Governance
May 27, 2026 · Big Data

10 Essential Data Cleaning Techniques Every AI Project Needs

The article outlines ten practical data‑cleaning methods—covering missing‑value imputation, duplicate handling, outlier detection, normalization, discretization, text cleaning, type conversion, multi‑source alignment, feature engineering, and sensitive‑data masking—explaining why each step matters for reliable AI model training.

big datadata cleaningdata masking
0 likes · 13 min read
10 Essential Data Cleaning Techniques Every AI Project Needs
Big Data Tech Team
Big Data Tech Team
May 25, 2026 · Big Data

AI Large Models Meet Data Warehouses: 3 Core Use Cases, 5 Common Pitfalls, and Best Practices

The article analyzes how AI large models can transform data‑warehouse development through three practical scenarios—automated modeling, intelligent data cleaning, and ops optimization—while exposing five frequent implementation traps and offering concrete best‑practice recommendations to achieve cost reduction, efficiency gains, and quality improvement.

AI large modelsAutomated ModelingBest Practices
0 likes · 10 min read
AI Large Models Meet Data Warehouses: 3 Core Use Cases, 5 Common Pitfalls, and Best Practices
Architect's Ambition
Architect's Ambition
May 18, 2026 · Artificial Intelligence

Building Enterprise Private Knowledge Bases: End-to-End Crawl, Clean, and RAG Pipeline

The article outlines a complete six‑stage workflow for constructing enterprise‑grade private knowledge bases—starting with targeted web‑crawling and API ingestion, through data cleaning, chunking, embedding generation, vector storage, and finally multi‑stage RAG retrieval optimization—highlighting why early stages set the performance ceiling and offering practical tips from real‑world projects.

AI agentEmbeddingRAG
0 likes · 10 min read
Building Enterprise Private Knowledge Bases: End-to-End Crawl, Clean, and RAG Pipeline
IT Services Circle
IT Services Circle
May 15, 2026 · Artificial Intelligence

Why Your Validation Set Fails: Outliers Are Skewing Your Data

The article explains how outliers can dramatically distort training and validation results in machine learning, outlines practical detection methods such as business rules, Z‑Score, IQR and Isolation Forest, and demonstrates cleaning techniques with a complete house‑price prediction case study in Python.

Isolation ForestPythondata cleaning
0 likes · 19 min read
Why Your Validation Set Fails: Outliers Are Skewing Your Data
Smart Workplace Lab
Smart Workplace Lab
May 10, 2026 · Artificial Intelligence

When Your Internal AI Is Fed Bad Data, How to Fix It?

The article recounts a real incident where an AI‑generated SOP cited outdated policy because a knowledge base was overloaded with unchecked historical documents, then outlines a step‑by‑step protocol—including corpus cleaning, version locking, and isolation zones—to prevent data contamination and ensure reliable AI outputs.

AIRAGdata cleaning
0 likes · 7 min read
When Your Internal AI Is Fed Bad Data, How to Fix It?
AI Architect Hub
AI Architect Hub
Apr 24, 2026 · Artificial Intelligence

RAG Level 1: Avoid Dirty Data Poisoning Your AI – A Data Cleaning Guide

This article explains why noisy documents cripple Retrieval‑Augmented Generation, enumerates common garbage data types, describes three typical data‑quality problems, warns against over‑cleaning, encoding, and regex pitfalls, and provides a configurable LangChain pipeline with deduplication and validation best practices.

AIEmbeddingLangChain
0 likes · 21 min read
RAG Level 1: Avoid Dirty Data Poisoning Your AI – A Data Cleaning Guide
AI Architect Hub
AI Architect Hub
Apr 19, 2026 · Artificial Intelligence

Mastering RAG: From Data Cleaning to Vector DBs in AI Applications

This article introduces the second stage of a large‑model application series, detailing the value of Retrieval‑Augmented Generation (RAG), its architecture, and a step‑by‑step outline covering data cleaning, text chunking, vectorization, vector‑DB selection, recall strategies, reranking, and prompt construction.

AILLMRAG
0 likes · 4 min read
Mastering RAG: From Data Cleaning to Vector DBs in AI Applications
Data Integration and Governance
Data Integration and Governance
Mar 19, 2026 · Fundamentals

Eight Essential Data Cleaning Techniques Every Analyst Should Know

The article outlines eight practical data cleaning methods—handling missing values, duplicate records, outliers, type conversion, standardization, inconsistent data, feature encoding, and text preprocessing—explaining why each step matters, how to apply specific techniques, and when to choose particular approaches.

Type Conversiondata cleaningduplicate removal
0 likes · 15 min read
Eight Essential Data Cleaning Techniques Every Analyst Should Know
Data Integration and Governance
Data Integration and Governance
Mar 18, 2026 · Fundamentals

Finally, a Clear Guide to Managing Data Quality End‑to‑End

The article explains why data quality is critical for reliable business decisions and walks through a complete end‑to‑end management framework—including lifecycle stages, organizational responsibilities, processes, tools, culture, and six practical methods such as cleaning, deduplication, standardization, validation, monitoring, and repair.

Data ManagementETLdata cleaning
0 likes · 12 min read
Finally, a Clear Guide to Managing Data Quality End‑to‑End
Fun with Large Models
Fun with Large Models
Feb 27, 2026 · Artificial Intelligence

Step‑by‑Step EasyDataset Workflow for Building High‑Quality LLM Training Data

This guide walks readers through installing EasyDataset, creating a project, uploading documents, choosing appropriate chunking strategies, cleaning the data, generating domain tag trees, and exporting a polished pre‑training dataset, with concrete examples, configuration screenshots, and practical recommendations for each step.

AI ModelEasyDatasetLLM data preparation
0 likes · 20 min read
Step‑by‑Step EasyDataset Workflow for Building High‑Quality LLM Training Data
Tech Musings
Tech Musings
Feb 7, 2026 · Fundamentals

How to Clean and Convert a Chinese Poetry Dataset for RAG Projects

This guide explains how to clean a Chinese poetry corpus—removing special characters, filtering short entries, and converting traditional characters to simplified Chinese—using Python validation functions, batch file processing, and WSL‑based OpenCC conversion, then persisting the results as JSON.

JSONRAGdata cleaning
0 likes · 12 min read
How to Clean and Convert a Chinese Poetry Dataset for RAG Projects
php Courses
php Courses
Jan 15, 2026 · Backend Development

Master PHP’s trim(): Clean Up Strings Efficiently

This guide explains why unwanted whitespace harms data quality, introduces PHP's trim() function and its variants, shows practical code examples for cleaning user input, CSV files, and API parameters, and offers performance tips and best‑practice recommendations for robust backend development.

PHPString ManipulationTRIM
0 likes · 9 min read
Master PHP’s trim(): Clean Up Strings Efficiently
Data Integration and Governance
Data Integration and Governance
Dec 24, 2025 · Fundamentals

Why Data Cleaning Can't Be Skipped: A Clear Guide to Avoid Common Pitfalls

Data cleaning is the foundation of any analysis; the article walks through profiling, handling missing values, removing duplicates, correcting outliers, standardizing formats, and linking tables, while illustrating each step with real‑world examples and warning against common shortcuts that waste time.

data cleaningdata preprocessingdata quality
0 likes · 10 min read
Why Data Cleaning Can't Be Skipped: A Clear Guide to Avoid Common Pitfalls
Engineer Tang's Implementation Study Notes
Engineer Tang's Implementation Study Notes
Dec 2, 2025 · Interview Experience

5-Year Implementation Engineer Interviews: Basic Linux & SQL Questions Revealing Skill Gaps

An experienced interviewer shares real Linux and SQL questions given to 5-year implementation engineer candidates, revealing wide skill gaps in basic commands like top, df, du, ps, netstat, find, and SQL joins, recursive queries, and data cleaning — with standard answers and troubleshooting workflows.

InterviewLinuxSQL
0 likes · 10 min read
5-Year Implementation Engineer Interviews: Basic Linux & SQL Questions Revealing Skill Gaps
IT Services Circle
IT Services Circle
Oct 15, 2025 · Fundamentals

Master Fuzzy String Matching in Python with fuzzywuzzy: A Practical Guide

Learn how to efficiently clean and deduplicate textual data using Python's fuzzywuzzy library, covering Levenshtein distance fundamentals, installation, three core matching functions, advanced process extraction, and real-world code examples for handling messy Chinese strings and standardizing company names.

LevenshteinPythondata cleaning
0 likes · 7 min read
Master Fuzzy String Matching in Python with fuzzywuzzy: A Practical Guide
DataFunSummit
DataFunSummit
Sep 22, 2025 · Artificial Intelligence

Explore Cutting-Edge AI-Driven Data Governance: A Comprehensive Resource Guide

This article presents a curated list of cutting‑edge topics covering AI‑powered data governance, large‑model applications, intelligent operations, and advanced analytics, offering readers a concise overview of emerging practices and case studies from industry leaders.

AIAnalyticsIntelligent Operations
0 likes · 2 min read
Explore Cutting-Edge AI-Driven Data Governance: A Comprehensive Resource Guide
DataFunTalk
DataFunTalk
Sep 15, 2025 · Artificial Intelligence

Unlocking the Future: AI-Driven Data Governance and Large Model Innovations

This article presents a curated catalog of cutting‑edge topics covering AI‑powered data governance, large‑model applications, data cleaning, compliance, lakehouse integration, intelligent operations, and generative analytics, inviting readers to explore the latest innovations and download the full e‑book via QR code.

AIAnalyticsIntelligent Operations
0 likes · 2 min read
Unlocking the Future: AI-Driven Data Governance and Large Model Innovations
Python Crawling & Data Mining
Python Crawling & Data Mining
Aug 26, 2025 · Fundamentals

Mastering Python Web Scraping: Clean Price Extraction with XPath Tricks

This article walks through a Python web‑scraping problem, demonstrates why the original XPath extraction returns noisy or missing price data, and provides multiple refined code solutions—including filtering empty entries, correcting XPath selectors, and using map‑filter techniques—to produce clean, formatted price lists.

Pythondata cleaninglxml
0 likes · 6 min read
Mastering Python Web Scraping: Clean Price Extraction with XPath Tricks
Data Practical Notes
Data Practical Notes
Jul 21, 2025 · Big Data

Six Data Cleaning Problems That Undermine Your Analysis

This article outlines six critical data cleaning challenges—standardization inconsistencies, missing and anomalous values, structural chaos, cross-system key mismatches, duplicate records, and ambiguous metric definitions—and provides practical resolution strategies for each to ensure reliable analytics.

data cleaningdata deduplicationdata quality
0 likes · 11 min read
Six Data Cleaning Problems That Undermine Your Analysis
Alibaba Cloud Observability
Alibaba Cloud Observability
Jul 14, 2025 · Cloud Native

How to Tame Massive New‑Energy‑Vehicle Logs with Cloud‑Native SLS Data Cleaning

This article explains the five major challenges of handling heterogeneous, weakly‑structured log data in the new‑energy‑vehicle ecosystem and demonstrates how Alibaba Cloud's Log Service (SLS) provides cloud‑native, real‑time data cleaning, cross‑region aggregation, cost‑optimized storage, and visual lineage to enable comprehensive operational insight and intelligent management.

SLSdata cleaninglog processing
0 likes · 18 min read
How to Tame Massive New‑Energy‑Vehicle Logs with Cloud‑Native SLS Data Cleaning
G7 EasyFlow Tech Circle
G7 EasyFlow Tech Circle
May 9, 2025 · Artificial Intelligence

How LLMs + Python Are Redefining Data Analysis: A Practical Guide

This article explains how large language models combined with Python's data‑science ecosystem can automate metadata extraction, data cleaning, and analysis tasks—illustrated with a step‑by‑step Titanic passenger dataset case study, complete prompts, code snippets, and best‑practice recommendations.

LLMPandasPython
0 likes · 18 min read
How LLMs + Python Are Redefining Data Analysis: A Practical Guide
php Courses
php Courses
May 7, 2025 · Fundamentals

Comprehensive Guide to Pandas Data Processing in Python

This tutorial provides a detailed overview of Pandas, covering its core data structures, data import/export, selection, cleaning, aggregation, merging, and a practical sales analysis example, with complete code snippets for each operation.

Pandasdata aggregationdata cleaning
0 likes · 8 min read
Comprehensive Guide to Pandas Data Processing in Python
Big Data Tech Team
Big Data Tech Team
Apr 21, 2025 · Industry Insights

8 Practical Ways DeepSeek Boosts Data Quality for Better Governance

This guide outlines eight concrete methods DeepSeek uses to improve data quality—including automated cleaning, validation, classification, monitoring, governance standards, anomaly detection, integration, and intelligent analysis—providing actionable steps for organizations to enhance data accuracy, completeness, consistency, and usability.

DeepSeekdata cleaningdata integration
0 likes · 5 min read
8 Practical Ways DeepSeek Boosts Data Quality for Better Governance
Sohu Tech Products
Sohu Tech Products
Apr 16, 2025 · Artificial Intelligence

Comprehensive Guide to Building AI Datasets: From Source Collection to Data Augmentation and Validation

This guide walks readers through every stage of building high‑quality AI training datasets—from locating open‑source data and defining goals, through collection, annotation, cleaning, large‑scale processing, optional augmentation, and splitting, to validation—using a medical QA example for fine‑tuning DeepSeek‑R1.

AI fine-tuningData AugmentationPython
0 likes · 18 min read
Comprehensive Guide to Building AI Datasets: From Source Collection to Data Augmentation and Validation
DaTaobao Tech
DaTaobao Tech
Mar 31, 2025 · Artificial Intelligence

AI Audio Generation and Voice Synthesis Practices at Taobao

The article surveys Taobao’s AI‑generated audio pipeline, detailing eight technical papers on image‑to‑video, OpenAI o1, multimodal video, and large‑model voice synthesis, while highlighting advances like VALL‑E, CosyVoice, F5‑TTS, data‑cleaning methods, and e‑commerce applications such as voice‑cloned live streams, multilingual TTS, AI video‑audio integration, and audiobook production.

AI AudioTTSdata cleaning
0 likes · 11 min read
AI Audio Generation and Voice Synthesis Practices at Taobao
Alibaba Cloud Native
Alibaba Cloud Native
Mar 25, 2025 · Cloud Native

Shift Data Cleaning Server‑Side with SPL: Boost Real‑Time Log Processing

Alibaba Cloud Log Service’s new SPL‑based rule consumption lets users move complex data‑cleaning logic from client code to the server, offering low‑code configuration, high performance, precise filtering, and significant reductions in latency, bandwidth, and compute resources across typical scenarios such as Python SDK processing and Flink integration.

Log ServiceSPLdata cleaning
0 likes · 11 min read
Shift Data Cleaning Server‑Side with SPL: Boost Real‑Time Log Processing
Test Development Learning Exchange
Test Development Learning Exchange
Feb 11, 2025 · Fundamentals

Master Data Cleaning in Pandas: 20 Essential Scripts for E‑Commerce Sales Analysis

This guide walks you through creating a sample e‑commerce sales Excel file with Pandas and then demonstrates twenty practical data‑cleaning and transformation scripts—including handling missing values, renaming columns, filtering, grouping, and exporting—so you can efficiently prepare sales data for analysis.

ExcelPandasdata analysis
0 likes · 9 min read
Master Data Cleaning in Pandas: 20 Essential Scripts for E‑Commerce Sales Analysis
Java Web Project
Java Web Project
Feb 5, 2025 · Big Data

Master DeepSeek: Install, Configure, and Harness Its Data Processing Power

This guide walks you through DeepSeek’s core capabilities—including installation on Windows, macOS, and Linux, configuration of storage paths, API keys, and logging levels, as well as data import, cleaning, analysis, visualization, batch processing, scheduling, and plugin extensions—providing concrete command examples and troubleshooting tips.

DeepSeekautomationcommand line
0 likes · 8 min read
Master DeepSeek: Install, Configure, and Harness Its Data Processing Power
Python Crawling & Data Mining
Python Crawling & Data Mining
Jan 28, 2025 · Fundamentals

Master Pandas: From Data Import to Advanced Manipulation in Python

This tutorial walks you through pandas fundamentals—including reading CSV/Excel files, creating Series and DataFrames, performing basic operations, cleaning data, using loc/iloc indexing, grouping, concatenating, merging, and handling time series—providing code examples and visual outputs for each step.

Time Seriesdata cleaninggroupby
0 likes · 14 min read
Master Pandas: From Data Import to Advanced Manipulation in Python
Baobao Algorithm Notes
Baobao Algorithm Notes
Nov 14, 2024 · Artificial Intelligence

How OpenCoder’s RefineCode Dataset Powers Next‑Gen Code LLMs

The OpenCoder technical report details the creation of the RefineCode dataset, its multi‑stage preprocessing, filtering, and sampling pipelines, the pre‑training and fine‑tuning schedules for 1.5B and 8B models, and the autonomous data selection methods that together achieve performance comparable to Qwen2.5‑Coder.

AutoDSCode LLMInstruction Tuning
0 likes · 18 min read
How OpenCoder’s RefineCode Dataset Powers Next‑Gen Code LLMs
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 24, 2024 · Artificial Intelligence

From Zero to One: A Practical Guide to Pretraining Large Language Models

This comprehensive guide walks you through every stage of LLM pretraining—from data sourcing, cleaning, and deduplication to tokenizer design, model architecture choices, training framework selection, optimization tricks, and evaluation methods—highlighting common pitfalls and practical solutions for building robust models.

LLM Pretrainingcurriculum learningdata cleaning
0 likes · 34 min read
From Zero to One: A Practical Guide to Pretraining Large Language Models
Python Crawling & Data Mining
Python Crawling & Data Mining
Aug 15, 2024 · Fundamentals

Master Pandas: Essential Data Manipulation Techniques for Beginners

This comprehensive tutorial walks you through pandas basics, including reading CSV and Excel files, creating Series and DataFrames, performing data inspection, cleaning, indexing, hierarchical indexing, time‑series handling, grouping, aggregation, concatenation, merging, and practical code examples with visual outputs.

Time Seriesdata cleaninggroupby
0 likes · 12 min read
Master Pandas: Essential Data Manipulation Techniques for Beginners
Test Development Learning Exchange
Test Development Learning Exchange
Jul 14, 2024 · Fundamentals

Using pandas fillna() to Handle Missing Data: 10 Practical Examples

This article introduces pandas' fillna() method and demonstrates ten practical examples—including basic filling, column‑specific values, forward/backward filling, limiting fills, using other DataFrames, functions, conditional fills, dictionaries, and Series—to help developers effectively handle missing data in Python data analysis.

PandasPythondata cleaning
0 likes · 6 min read
Using pandas fillna() to Handle Missing Data: 10 Practical Examples
Ops Development & AI Practice
Ops Development & AI Practice
Jul 8, 2024 · Artificial Intelligence

Essential Denoising Techniques for Training Large AI Models

This article outlines key denoising methods—including data cleaning, augmentation, regularization, adversarial training, and self‑supervised learning—that improve the performance, generalization, and robustness of large neural network and transformer models.

Data AugmentationDenoisingRegularization
0 likes · 5 min read
Essential Denoising Techniques for Training Large AI Models
Test Development Learning Exchange
Test Development Learning Exchange
May 21, 2024 · Artificial Intelligence

Step-by-Step Data Analysis and Machine Learning Workflow with Pandas, Matplotlib, and Scikit-learn

This guide walks through loading CSV data with pandas, cleaning missing values, filtering, grouping, visualizing, performing correlation and time‑series analysis, detecting outliers, and applying linear and logistic regression models using scikit‑learn, all illustrated with complete Python code snippets.

Pandasdata cleaningmachine learning
0 likes · 6 min read
Step-by-Step Data Analysis and Machine Learning Workflow with Pandas, Matplotlib, and Scikit-learn
Python Programming Learning Circle
Python Programming Learning Circle
Apr 28, 2024 · Fundamentals

Data Cleaning Techniques in Python: 21 Practical Examples and Code

This tutorial explains data cleaning concepts, key quality dimensions, and demonstrates 21 practical Python examples—including regex phone cleaning, temperature conversion, missing‑value detection, visualization with missingno, and record linkage using fuzzy matching—providing clear code snippets and step‑by‑step guidance for reliable data analysis.

Pandasdata cleaningmissing data
0 likes · 20 min read
Data Cleaning Techniques in Python: 21 Practical Examples and Code
Model Perspective
Model Perspective
Feb 13, 2024 · Big Data

Mastering Noisy Data: From Cleaning to Visualization and NLP with Python

This article reviews the key concepts from the Bad Data Handbook, covering noise identification, data validation, human readability, web data restructuring, special domain challenges, and data quality analysis, while also presenting practical data visualization techniques, popular analysis tools, Python web‑scraping libraries, and a basic NLP workflow with code examples.

NLPPythondata cleaning
0 likes · 20 min read
Mastering Noisy Data: From Cleaning to Visualization and NLP with Python
Python Programming Learning Circle
Python Programming Learning Circle
Feb 4, 2024 · Fundamentals

Using FuzzyWuzzy for Fuzzy String Matching in Python

This article introduces the FuzzyWuzzy Python library, explains its Levenshtein‑based matching functions (Ratio, Partial Ratio, Token Sort Ratio, Token Set Ratio) and the process module, and demonstrates practical applications for fuzzy matching of company and province names with complete code examples.

Levenshtein distancePandasPython
0 likes · 10 min read
Using FuzzyWuzzy for Fuzzy String Matching in Python
Test Development Learning Exchange
Test Development Learning Exchange
Dec 4, 2023 · Fundamentals

Common Data Cleaning Techniques with Python Code Examples

This article presents a comprehensive collection of Python code snippets demonstrating essential data cleaning methods—including handling missing values, outlier detection, type conversion, formatting, duplicate removal, normalization, one‑hot encoding, text preprocessing, and dataset merging—providing practical guidance for preparing data for analysis or machine‑learning tasks.

Pandasdata cleaningdata preprocessing
0 likes · 7 min read
Common Data Cleaning Techniques with Python Code Examples
Python Programming Learning Circle
Python Programming Learning Circle
Jun 7, 2023 · Fundamentals

Using FuzzyWuzzy for Fuzzy String Matching in Python

This article introduces the FuzzyWuzzy Python library, explains its Levenshtein‑based matching functions such as Ratio, Partial Ratio, Token Sort Ratio and Token Set Ratio, demonstrates how to install it, and provides practical code examples for merging company and province fields with fuzzy matching thresholds.

Pandasdata cleaningstring matching
0 likes · 10 min read
Using FuzzyWuzzy for Fuzzy String Matching in Python
Zhuanzhuan Tech
Zhuanzhuan Tech
May 26, 2023 · Backend Development

ECP (Elasticsearch Chain Planning) System: Design, Features, and Implementation for Efficient Index Management

The article introduces the ECP system, a backend platform built on Elasticsearch that standardizes, automates, and visualizes index refresh workflows, addressing manual bottlenecks, data cleaning challenges, and coupling issues while providing task management, permission control, and environment isolation for high‑efficiency index operations.

Elasticsearchautomationdata cleaning
0 likes · 12 min read
ECP (Elasticsearch Chain Planning) System: Design, Features, and Implementation for Efficient Index Management
Python Programming Learning Circle
Python Programming Learning Circle
Mar 31, 2023 · Fundamentals

Vectorized String Operations in Pandas: Methods and Examples

This article explains how Pandas' vectorized string operations enable efficient, loop‑free processing of text data, covering basic methods like len() and lower(), advanced regex functions, and additional utilities such as split, replace, slice, and get_dummies, with code examples and usage details.

String Processingdata cleaningvectorization
0 likes · 21 min read
Vectorized String Operations in Pandas: Methods and Examples
Python Programming Learning Circle
Python Programming Learning Circle
Dec 10, 2022 · Fundamentals

Using Python (pandas) to Perform Common Excel Data Processing Tasks

This article demonstrates how to replace typical Excel operations such as VLOOKUP, pivot tables, duplicate removal, missing‑value handling, multi‑condition filtering, fuzzy matching, column splitting, outlier replacement, grouping and labeling with concise Python pandas code to streamline data analysis workflows.

VLOOKUPdata cleaningdata processing
0 likes · 9 min read
Using Python (pandas) to Perform Common Excel Data Processing Tasks
Python Programming Learning Circle
Python Programming Learning Circle
Aug 13, 2022 · Big Data

Parallel Processing of Large CSV Files in Python Using multiprocessing, joblib, and tqdm

This tutorial demonstrates how to accelerate processing of a multi‑million‑row CSV dataset by splitting the work into sub‑tasks and applying Python's multiprocessing, joblib, and tqdm libraries for serial, parallel, and batch processing, showing significant speed‑ups and best‑practice code snippets.

Parallel ProcessingPythonbig data
0 likes · 10 min read
Parallel Processing of Large CSV Files in Python Using multiprocessing, joblib, and tqdm
Model Perspective
Model Perspective
Jun 3, 2022 · Fundamentals

Master Pandas: From Installation to Data Analysis in Python

This tutorial walks you through installing Pandas, importing data from CSV and Excel, creating DataFrames from dictionaries, describing datasets, indexing with loc/iloc, and cleaning columns using apply, all illustrated with clear code examples and visual outputs.

data cleaningdata importdata-analysis
0 likes · 6 min read
Master Pandas: From Installation to Data Analysis in Python
DataFunTalk
DataFunTalk
May 25, 2022 · Artificial Intelligence

Optimizing E-commerce Product Copy Generation: Challenges, Framework, and System Practices

This article presents a comprehensive overview of the challenges in e‑commerce product copy generation, introduces a unified framework comprising a copy generation system, a copy‑cleaning subsystem, and a quality evaluation module, and details practical optimization techniques applied to short and long copy scenarios.

AIdata cleaningmodel optimization
0 likes · 17 min read
Optimizing E-commerce Product Copy Generation: Challenges, Framework, and System Practices
Python Crawling & Data Mining
Python Crawling & Data Mining
Apr 27, 2022 · Fundamentals

Master Pandas: From CSV to Advanced Data Manipulation in Python

This comprehensive tutorial walks you through pandas fundamentals—including reading CSV/Excel files, creating Series and DataFrames, performing basic operations, cleaning data, indexing, grouping, concatenation, merging, and handling time series—using clear examples and code snippets.

Data ManipulationPandasdata analysis
0 likes · 12 min read
Master Pandas: From CSV to Advanced Data Manipulation in Python
Laravel Tech Community
Laravel Tech Community
Mar 29, 2022 · Databases

Efficient Methods for Removing Duplicate Records in MySQL Tables

This article explains why a naïve Python‑based row‑deletion approach is slow for large MySQL tables and provides step‑by‑step SQL techniques—including identifying duplicate names, handling MySQL’s update‑from‑same‑table limitation, and deleting duplicates while preserving a single record per group—complete with executable code examples.

SQLdata cleaningduplicate removal
0 likes · 5 min read
Efficient Methods for Removing Duplicate Records in MySQL Tables