Unsupervised Community Detection for Black‑Market Identification Using the Louvain Algorithm

In the current thriving Internet environment, enterprises constantly face black‑ and gray‑market operators who exploit anonymity, widespread technology, and cross‑regional connectivity, leading to an ecological, intelligent, and precise black‑market ecosystem that poses unprecedented risks to online governance.

The paper first outlines the characteristics of black‑market activity—latency, group behavior, predation, and lack of restriction—and reviews existing detection methods: content‑based filtering (image, text, audio‑video) and behavior‑based modeling of accounts, noting their limitations for hidden or large‑scale threats.

A five‑layer system architecture is then introduced: a data‑source layer that aggregates multimedia, basic, and behavioral data via Kafka or Hive; a relationship‑construction layer that builds device, resource, behavior, and inference graphs; an algorithm‑computation layer that employs Spark to construct graphs and extract sub‑graphs for community detection; a data‑output layer that stores results in Hive, returns DataFrames, or caches in Redis for real‑time queries; and a scenario‑application layer that uses detected communities for risk recall, user and resource auditing.

The core detection technique is the Louvain algorithm, a modularity‑based community‑discovery method. Modularity Q measures the density of intra‑community edges versus a random baseline, and the algorithm iteratively moves nodes to communities that maximize the increase in Q. The basic workflow is shown below:

1. Initialize each node in its own community
2. Iterate over nodes, assigning each to the community that yields the greatest modularity gain
3. Merge nodes belonging to the same community into a super‑node
4. Rebuild the graph with super‑nodes
5. Repeat steps 2‑4 until convergence

To address the tendency of the original Louvain method to merge small communities into overly large ones, an optimization step is added between steps 3 and 4 to prune low‑weight edges:

3.1 Check if edge weight in the new graph falls below a threshold
3.2 Cut edges with weight below the threshold (set weight to 0)

Experimental results on a 58.com business dataset—using phone numbers, WeChat IDs, IPs, device IDs, and other identifiers—show that the raw Louvain algorithm clusters most users as normal, with dense red regions indicating suspicious groups. After optimization, large communities are split, increasing the proportion of risky users within each community and improving detection value.

Sampling of a highlighted suspicious group reveals that over 95% of sampled accounts are malicious, confirming the effectiveness of the approach. In production, custom metrics such as community size, weight, and resource‑share are used to filter groups, achieving an average identification accuracy above 70% and a recall increase of more than 30% compared with previous strategies.

The paper concludes with two future plans: (1) integrating additional algorithmic models to handle diverse business scenarios, and (2) incorporating richer behavioral features to address cold‑start problems when newly created accounts lack relational edges.

Reference: "Fast unfolding of communities in large networks" (arXiv:0803.0476).

Author: Zhuang Wei, Senior Algorithm Engineer, Information Security Department, 58.com, responsible for risk‑control algorithm design and development since 2019.