Tagged articles

text deduplication

4 articles · Page 1 of 1

Jul 30, 2026 · Big Data

How EMR Serverless Spark Achieves 4× Faster PB‑Scale Text Deduplication

The article analyzes how migrating a large‑scale text deduplication workflow to Alibaba Cloud EMR Serverless Spark, using built‑in MinHash‑LSH functions and the Fusion Engine vectorized executor, reduces processing time from days to hours, cuts shuffle failures to zero, and eliminates most operational overhead.

AI data preprocessingEMR Serverless SparkFusion Engine

0 likes · 16 min read

How EMR Serverless Spark Achieves 4× Faster PB‑Scale Text Deduplication

Qunhe Technology Quality Tech

Dec 18, 2025 · Artificial Intelligence

How We Slashed AI Token Costs by Up to 90% with Smart Pipeline Optimizations

This report details a systematic analysis of AI token consumption in a multilingual UI‑automation workflow and presents four concrete optimization techniques—prompt trimming, duplicate‑call avoidance, text deduplication, and placeholder‑based knowledge‑base integration—that together reduced monthly token usage by over 90% without harming detection accuracy.

AI token optimizationautomation pipelinecost reduction

0 likes · 14 min read

How We Slashed AI Token Costs by Up to 90% with Smart Pipeline Optimizations

Sohu Smart Platform Tech Team

Aug 9, 2025 · Artificial Intelligence

How SimHash and Cosine Similarity Accelerate Large-Scale Text Deduplication

This article explains why traditional pairwise text comparison is impractical for massive news corpora, introduces cosine similarity and SimHash as efficient deduplication techniques, walks through their mathematical foundations, step‑by‑step implementation details, code examples, and discusses trade‑offs such as accuracy versus speed.

SimHashalgorithmbig data

0 likes · 12 min read

How SimHash and Cosine Similarity Accelerate Large-Scale Text Deduplication

Sohu Tech Products

Feb 28, 2024 · Big Data

How SimHash and Cosine Similarity Accelerate Large‑Scale Text Deduplication

This article explains why massive news feeds need efficient deduplication, compares cosine similarity and SimHash for measuring text similarity, walks through a step‑by‑step implementation with Java code, and shows how a space‑for‑time indexing strategy can reduce duplicate‑detection complexity from O(n²) to near O(1).

Near-Duplicate DetectionSimHashalgorithm

0 likes · 14 min read