Tagged articles

9 articles

Page 1 of 1

Aug 9, 2025 · Artificial Intelligence

How SimHash and Cosine Similarity Accelerate Large-Scale Text Deduplication

This article explains why traditional pairwise text comparison is impractical for massive news corpora, introduces cosine similarity and SimHash as efficient deduplication techniques, walks through their mathematical foundations, step‑by‑step implementation details, code examples, and discusses trade‑offs such as accuracy versus speed.

Big DataCosine SimilaritySimHash

0 likes · 12 min read

How SimHash and Cosine Similarity Accelerate Large-Scale Text Deduplication

Baobao Algorithm Notes

Oct 25, 2024 · Artificial Intelligence

How Simhash and Minhash Power LLM Data Deduplication: Theory and Spark Code

This article explains document‑level, paragraph‑level, and sentence‑level deduplication for large‑scale LLM pre‑training, introduces the Simhash and Minhash algorithms with step‑by‑step Python examples, and shows how to implement efficient LSH‑based deduplication using Spark.

LLMMinhashPython

0 likes · 29 min read

How Simhash and Minhash Power LLM Data Deduplication: Theory and Spark Code

ZhongAn Tech Team

Sep 3, 2024 · Big Data

Real-Time Log Clustering Architecture and Continuous Clustering Algorithm

This article presents a comprehensive overview of a log clustering system, detailing its background, architecture based on Filebeat, Kafka, Flink, Elasticsearch, and Grafana, and introduces a continuous clustering algorithm using SimHash and Hamming distance for real‑time log governance and anomaly detection.

FlinkLog ClusteringReal-time analytics

0 likes · 14 min read

Real-Time Log Clustering Architecture and Continuous Clustering Algorithm

Sohu Tech Products

Feb 28, 2024 · Big Data

How SimHash and Cosine Similarity Accelerate Large‑Scale Text Deduplication

This article explains why massive news feeds need efficient deduplication, compares cosine similarity and SimHash for measuring text similarity, walks through a step‑by‑step implementation with Java code, and shows how a space‑for‑time indexing strategy can reduce duplicate‑detection complexity from O(n²) to near O(1).

Big DataCosine SimilarityNear-Duplicate Detection

0 likes · 14 min read

NetEase Cloud Music Tech Team

Mar 31, 2022 · Industry Insights

How Implicit Relationship Chains Solve Cold‑Start Problems at NetEase Cloud Music

This article details NetEase Cloud Music's technical approach to building implicit user relationship chains—using SimHash, Item2Vec, and MetaPath2Vec embeddings, large‑scale vector search, and a unified service architecture—to address cold‑start challenges across multiple business scenarios.

Item2VecMetaPath2VecRecommendation Systems

0 likes · 20 min read

How Implicit Relationship Chains Solve Cold‑Start Problems at NetEase Cloud Music

Architect

Oct 18, 2021 · Fundamentals

Understanding Simhash: From Traditional Hash to Random Projection and LSH

This article explains the principles behind Simhash, covering the shortcomings of traditional hash functions, the use of cosine similarity, random projection for dimensionality reduction, locality‑sensitive hashing, random hyperplane hashing, implementation steps, query optimization with the pigeonhole principle, and the algorithm's limitations in short‑text scenarios.

Locality Sensitive HashingRandom ProjectionSimHash

0 likes · 18 min read

Understanding Simhash: From Traditional Hash to Random Projection and LSH

Sohu Tech Products

Mar 17, 2021 · Big Data

Understanding Simhash: From Traditional Hash to Random Projection LSH

This article explains the principles and implementation of Simhash, covering the shortcomings of traditional hash functions, the use of cosine similarity, random projection for dimensionality reduction, locality‑sensitive hashing, and practical optimizations for large‑scale duplicate detection.

Big DataCosine SimilarityLocality Sensitive Hashing

0 likes · 24 min read

Understanding Simhash: From Traditional Hash to Random Projection LSH

360 Quality & Efficiency

Oct 19, 2018 · Big Data

Information Fingerprint and Simhash Algorithm for Large-Scale Duplicate Detection

This article explains the concept of information fingerprints, compares traditional set‑equality methods, introduces the Simhash algorithm for high‑dimensional text similarity reduction, and demonstrates how partitioned 64‑bit fingerprints enable efficient duplicate detection on massive web data.

Big DataSimHashduplicate detection

0 likes · 6 min read

Information Fingerprint and Simhash Algorithm for Large-Scale Duplicate Detection

Architecture Digest

Nov 17, 2016 · Big Data

Spam Detection on Zhihu Using Text and Behavior Clustering with Jaccard and SimHash on Spark

This article describes how Zhihu combats large‑scale spam by applying text and behavior clustering techniques—using Jaccard similarity, SimHash fingerprinting, and Spark‑based graph partitioning—to efficiently identify and group similar spammy content and actions.

Big DataSimHashclustering

0 likes · 11 min read

Spam Detection on Zhihu Using Text and Behavior Clustering with Jaccard and SimHash on Spark