Community Encoding Based Detection of Black and Gray Market Attacks Using Graph Embedding

1. Background

The so‑called black‑gray market includes both black‑market and gray‑market supply chains, which have evolved into platform‑based, professional, and fine‑grained operations as the Internet expands. These activities now encompass a multi‑billion‑RMB market with diverse sub‑domains such as malware, account farming, fraud, piracy, and traffic hijacking.

Typical defenses on platforms include captchas, rule‑engine filtering, behavior‑sequence modeling, graph‑based relationship mining, and various clustering techniques, each offering varying degrees of detection.

To address the specific characteristics of black‑gray attacks, we propose a community‑encoding detection method that combines graph‑relationship discovery with large‑scale graph‑embedding representation learning, improving identification of unknown attacks and offering an asynchronous near‑real‑time engineering implementation.

2. Community Structure

The method builds on graph mining by introducing massive graph‑embedding techniques, enabling extraction of both explicit and latent community structures for more accurate detection.

Two types of graphs are considered: homogeneous graphs, where all nodes share the same type (e.g., user account IDs), and heterogeneous graphs, where nodes can represent accounts, IPs, devices, phone numbers, etc.

Illustrative diagrams show the difference between homogeneous and heterogeneous graphs, as well as node attribute tables (e.g., activity time, business scenario, operation type).

In practice, we treat all graphs as heterogeneous, since homogeneous graphs are a special case of heterogeneous ones.

Existing community‑identification methods rely on node statistics, degree distribution, custom edge weights, or manual labeling, but they often suffer from false positives due to ambiguous edge weights. Our approach encodes discovered communities to enhance detection, and the embedding can learn similarity in an unsupervised manner.

3. Graph‑Embedding Encoding

We adopt GraphSAGE (Hamilton, Ying, and Leskovec, 2016) as the node‑embedding technique, which aggregates neighbor information at the graph level, offering better scalability and performance than node2vec.

GraphSAGE workflow:

Sample a fixed number of neighbors for each node.

Aggregate neighbor features using a chosen aggregation function.

Produce vector representations for nodes to be used in downstream tasks.

Sampling and aggregation are illustrated in the accompanying diagrams. In our implementation we use two sampling layers with a maximum of 200 neighbors per layer, suitable for large‑scale datasets.

We employ mean aggregation, concatenating the target node’s previous‑layer vector with its neighbors’ vectors, averaging each dimension, and applying a non‑linear transformation to obtain the next‑layer representation.

4. Model Training

Using the sampled and aggregated embeddings, we train an unsupervised loss that pulls neighboring nodes together and pushes distant nodes apart. The loss can be replaced by supervised objectives (e.g., cross‑entropy) for specific classification tasks.

Positive and negative samples are manually labeled based on node statistics; the resulting embeddings feed a classification model whose visualization shows distinct community clusters (red, yellow, green).

5. Engineering Implementation

We design an asynchronous near‑real‑time detection pipeline. Incoming client requests generate factor logs (account ID, IP, device ID, phone number, etc.) stored in a short‑term buffer (e.g., 10 minutes). Older logs are moved to an offline partitioned log store for graph construction, community encoding, and model training.

For requests still in the buffer, a real‑time graph is built around the user, sampled, embedded, and evaluated by the pre‑trained classifier. Normal users proceed; detected black‑gray users are blocked.

Detailed training and prediction flowcharts are provided in the figures.

6. Innovations

Supervised community‑encoding that captures whole‑community structure rather than isolated node attributes, enabling detection of previously unseen malicious accounts.

Embedding vectors naturally encode strong similarity among neighbors and weak similarity among non‑neighbors, allowing unsupervised clustering to reveal hidden relationships.

Combination of large‑scale graph‑embedding with a small‑data asynchronous prediction buffer for practical deployment.

7. Practical Results

The factor‑encoding model includes five key factors (IP, cookie_id, device_id, user_id, mobile) with additional context features (scene, page, risk). Using 10‑minute windows, we process millions of factors, generating tens of millions of relationships and ~300‑dimensional vectors.

Manual verification shows the supervised risk scoring for the IP dimension achieves ~95 % accuracy, while unsupervised models performed poorly and were not adopted.

8. Future Considerations

Challenges remain:

High computational cost of graph processing, especially for batch predictions.

Critical importance of factor selection and feature engineering for graph quality.

Partial automation of community definition; current pipelines still require manual or auxiliary modeling input.