Tagged articles
data deduplication
20 articles · Page 1 of 1
This article analyzes five root causes of data duplication, classifies duplicate types, compares SQL deduplication methods (DISTINCT, GROUP BY, ROW_NUMBER), covers source-level prevention with unique constraints and idempotency, explains real-time CDC deduplication strategies, and outlines a five-step process for building sustainable data quality governance.
CDCETLIdempotency
0 likes · 18 min read

The author used an LLM to automate discovery, deduplication, validation, and reporting of OSS domain migrations across 10+ databases, 1,000+ tables, and 100M+ records, completing in ~5 hours what would have taken a team days.
AI-assisted developmentAPI validationDoubao
0 likes · 11 min read

Learn how PHP’s built-in array_unique() function efficiently removes duplicate values from arrays, with detailed syntax, practical code examples, and real-world scenarios such as data cleaning, user input validation, and data aggregation to improve code readability and performance.
PHParray_uniquedata deduplication
0 likes · 7 min read

This article outlines six critical data cleaning challenges—standardization inconsistencies, missing and anomalous values, structural chaos, cross-system key mismatches, duplicate records, and ambiguous metric definitions—and provides practical resolution strategies for each to ensure reliable analytics.
data cleaningdata deduplicationdata quality
0 likes · 11 min read

This article explains how PHP’s built‑in array_unique() function works, shows syntax and code examples, and demonstrates practical scenarios such as data cleaning, user input validation, and re‑indexing arrays to efficiently remove duplicate values.
Code ExamplesPHParray_unique
0 likes · 6 min read

This article explains document‑level, paragraph‑level, and sentence‑level deduplication for large‑scale LLM pre‑training, introduces the Simhash and Minhash algorithms with step‑by‑step Python examples, and shows how to implement efficient LSH‑based deduplication using Spark.
LLMMinhashPython
0 likes · 29 min read

This article explains PHP’s built-in array_unique() function, detailing its syntax, behavior, and flags, and demonstrates how to remove duplicate values from arrays with practical code examples and real-world use cases such as data cleaning, input validation, and aggregation.
Backendarray_uniquearrays
0 likes · 6 min read

This article walks through a real‑world Python big‑data deduplication challenge, compares several memory‑efficient strategies—including tuple‑set, merge‑union, and concat‑drop_duplicates approaches—and offers practical tips for asking technical questions about large datasets.
big datadata deduplicationmemory optimization
0 likes · 3 min read

This article walks through practical techniques for removing duplicates from massive Python datasets—such as using tuples with sets, merging iteratively, and concatenating with drop_duplicates—while highlighting memory pitfalls and offering concise code snippets.
big datadata deduplicationlarge datasets
0 likes · 3 min read

This article explains how Renliji's data warehouse team leveraged MaxCompute's Transaction Table2.0 to dramatically reduce incremental data deduplication costs and execution time, while also introducing efficient small‑file merging, time‑travel queries, and future data‑sync strategies for a high‑growth HR SaaS platform.
Cost OptimizationMaxComputeTime Travel
0 likes · 11 min read

The article explains how space‑efficient probabilistic structures such as BloomFilter and Count‑min Sketch enable large‑scale data deduplication, join pruning, real‑time idempotent filtering, and approximate top‑K analytics by trading modest accuracy loss for dramatically reduced storage and faster computation.
BloomFilterCount-Min SketchHash
0 likes · 12 min read

This article explains why duplicate rows appear in MySQL tables, demonstrates how to identify them with SELECT queries, and provides step‑by‑step SQL solutions—including safe deletion of all duplicates or retaining a single record per group—using subqueries and temporary tables for efficient cleanup.
Duplicate DataMySQLSQL
0 likes · 5 min read

This article explains why duplicate rows appeared in production MySQL tables, demonstrates how to identify them with SELECT queries, and provides two SQL solutions—one to delete all duplicates and another to keep a single record per duplicated key—while preserving data integrity.
MySQLSQLdata deduplication
0 likes · 5 min read

This article explains why a naïve Python script for deleting duplicate MySQL rows is too slow, demonstrates the MySQL error caused by deleting from the same table you query, and provides two pure‑SQL solutions: one that removes all duplicates and another that keeps a single row per duplicate key.
MySQLSQLdata deduplication
0 likes · 5 min read

Data deduplication eliminates redundant data blocks to lower storage and bandwidth costs, while source‑ and transmission‑level encryption safeguards data in transit and at rest; the article also compares hardware vs software deduplication, various storage architectures (DAS, SAN, NAS, object and distributed storage) and their trade‑offs.
NASSANbackup
0 likes · 15 min read

This article explains how to identify and remove various kinds of duplicate data in MySQL—including fully duplicated rows, records with duplicate non‑key columns, and unwanted whitespace inside fields—by using SQL statements, table cloning, OS utilities, and regular‑expression updates, with performance measurements for each method.
Database MigrationMySQLSQL
0 likes · 13 min read

This article presents a Python script that merges multiple CSV files from a specified directory and removes duplicate rows using pandas, providing a practical solution for test case management.
Pythoncsv-processingdata deduplication
0 likes · 3 min read

This article explains why Spark Streaming combined with Kafka can only guarantee at‑least‑once delivery, outlines the challenges of delayed and out‑of‑order events, and presents practical offline‑repair, deduplication, and output‑format techniques—including code examples—to achieve exact‑once semantics in big‑data pipelines.
Exact-OnceHBaseHDFS
0 likes · 11 min read

This article provides a detailed technical analysis of Huawei's Dorado V3 all‑flash storage system, explaining its classification, the GRIP (Granular management, ROW, Inline deduplication & compression, Parity RAID) and FAST (FlashLink, Active‑active, Zero‑loss snapshot, RAID‑TP) technologies, and why these features are essential for modern flash‑oriented solutions.
All-flash storageDorado V3Flash Optimization
0 likes · 12 min read

This article provides an in‑depth, systematic overview of flash storage, covering architecture, metadata management, deduplication, wear‑leveling, power‑loss protection, NAND flash cell types, reliability techniques, emerging 3D‑Flash and memristor technologies, as well as PCIe/NVMe interface standards.
3D NANDNAND flashNVMe
0 likes · 20 min read
