0 views collected around this technical thread.
Learn how to use PHP’s array_unique() function to eliminate duplicate values from arrays, understand its parameters—including optional sort flags—and see practical code examples that demonstrate deduplication and sorting for both string and numeric arrays.
This article explains how to efficiently remove duplicates from 4 billion QQ numbers within a 1 GB memory limit by replacing the naïve HashSet approach with a memory‑saving Bitmap data structure, complete with calculations, algorithm steps, Java code, and a discussion of its pros and cons.
Message queues can cause duplicate messages that affect idempotent business processes, so this article explains the three delivery semantics (At Least Once, Exactly Once, At Most Once), the causes of duplication, and practical deduplication techniques for producers, brokers (Kafka, Pulsar), and consumers using code examples.
The article explains how to store and deduplicate billions of identifiers by using a bitmap to represent presence with a single bit per value, calculates memory requirements, shows Redis bitmap commands, and introduces Bloom filters as an extension with multiple hash functions for efficient large‑scale data handling.
This article explains PHP's array_unique() function, covering its definition, parameters, implementation, usage examples, and performance optimization techniques, while providing complete code snippets and practical guidance for developers, including discussion of alternative approaches such as array_flip and array_keys for faster deduplication.
This article explains Python's set type, covering its definition, creation (including empty sets), key properties such as unordered uniqueness, conversion using set(), the eight basic and four augmented set operators, common set methods, and practical applications like data deduplication.
This article explains PHP's array_unique() function, covering its definition, parameters, implementation details, usage examples, and performance optimization techniques for efficient array deduplication.
This article presents a comprehensive overview of precise deduplication in Meituan's Doris database, detailing the underlying bitmap data structures, aggregation bottlenecks, and a series of optimizations—including memory management, fast union, orthogonal encoding, and vectorized engine integration—that together achieve significant performance gains in high‑cardinality scenarios.
This article explains PHP’s array_unique() function, detailing its syntax, parameters, return value, and optional sorting flag, and provides code examples that demonstrate how to remove duplicate elements from arrays and control sorting behavior.
This article explains core concepts and practical solutions for handling massive datasets that cannot fit into memory, covering batch processing, distributed sorting, bitmap indexing, hash‑based lookups, top‑K extraction, and deduplication techniques with code examples and multi‑machine strategies.
This article explains Python's set data type, covering its definition, unordered and unique characteristics, creation methods, basic and augmented operators, practical usage examples, handling techniques, and common application scenarios such as data deduplication.
This article examines several Java deduplication techniques—including List.contains, HashSet, double-loop removal, and Stream.distinct—by providing sample code, measuring execution time on a 20,000‑element list, and analyzing their time complexities to guide developers toward efficient duplicate‑removal strategies.
The article explains how to store and de‑duplicate 4 billion unsigned integers using a bitmap to reduce memory from 14.9 GB to under 500 MB, introduces the concept and benefits of bitmaps, describes Bloom filters, their principles, advantages, limitations, typical use cases, and provides Java and Redis implementation examples.
This article presents an efficient deduplication algorithm for parent‑child relationship validation that caches intermediate results in a hashset to eliminate redundant computations, dramatically improving performance and scalability for large datasets by reducing verification steps through stored validated nodes.
This article explains various techniques for detecting and preventing duplicate backend requests—using unique request IDs, business parameter hashing, MD5 summaries, and Redis SETNX with expiration—providing Java code examples and a complete deduplication utility.
The article examines Tinode‑based message‑processing bottlenecks in a multi‑channel customer‑service dashboard and proposes a suite of optimizations—including global deduplication maps, binary‑search insertion sorting, cache reclamation, targeted status updates, and asynchronous keyword interception—that together cut average first‑response time from 8.40 s to 6.82 s and overall response time from 19.9 s to 18.22 s, proving that careful cache design and algorithmic refinements markedly boost real‑time IM performance.
This article explains three common SQL deduplication techniques—using DISTINCT, GROUP BY, and the ROW_NUMBER window function—illustrating their syntax, performance considerations, and example queries for counting unique task identifiers in relational databases.
The paper proposes a Bloom‑filter‑based deduplication service for short‑video recommendation that moves three‑month playback histories to disk‑backed Bloom filters while keeping the latest 100 served IDs in Redis, employing write batching, sharding, expiration policies, and an incremental migration strategy to replace memory‑intensive Redis ZSets and dramatically reduce storage costs.
This article explains the challenges of at‑least‑once delivery in distributed message middleware like RocketMQ, examines simple and concurrent deduplication techniques, and presents both transaction‑based and non‑transactional exactly‑once solutions using database tables or Redis, along with practical Java code examples.
This article explains how to remove duplicate rows in SQL by using DISTINCT, GROUP BY, and the ROW_NUMBER window function, compares their syntax and performance, and provides concrete examples with a Task table and a Test helper table.