URL normalization

Shopee Tech Team

Apr 14, 2022 · Big Data

URL Normalization and Statistical Analysis in MDAP Using Probabilistic and Machine Learning Techniques

MDAP normalizes URLs by automatically learning pattern‑tree rule models using entropy‑based splits, gibberish and numeric detection, and scalable Flink processing, which groups millions of raw URLs into concise patterns for accurate statistical monitoring, dramatically reducing data noise while still facing latency and model‑iteration challenges.