Finding Common URLs in Billions of Records: From Naive Search to Hash Divide‑and‑Conquer

This article walks through a classic interview problem of locating common URLs in two massive files, explains why a naïve O(m·n) approach fails, and presents a scalable solution using hash‑based divide‑and‑conquer and file partitioning techniques.

Big Datadivide and conquerhash algorithm

0 likes · 6 min read

Finding Common URLs in Billions of Records: From Naive Search to Hash Divide‑and‑Conquer

Full-Stack Internet Architecture

Sep 13, 2020 · Backend Development

URL Deduplication Techniques in Java, Redis, and Databases

This article reviews six practical URL deduplication methods—including Java Set, Redis Set, database queries, unique indexes, Guava Bloom filter, and Redis Bloom filter—explaining their principles, providing complete implementation code, and recommending the most suitable approach for different system scales.

Setbloom-filterdatabase

0 likes · 13 min read

URL Deduplication Techniques in Java, Redis, and Databases

Sohu Tech Products

Dec 5, 2018 · Backend Development

Overview of Web Crawler Types and the Architecture of the Mole Crawler System

This article explains the evolution and classification of web crawlers, describes the design and components of the Mole distributed crawler—including scheduler, fetcher, processor, rate‑limiting, URL deduplication, and Elasticsearch storage optimization—and outlines common anti‑anti‑crawling strategies.

ElasticsearchWeb Crawleranti‑crawling

0 likes · 12 min read

Overview of Web Crawler Types and the Architecture of the Mole Crawler System