Seeing Inside the Black Box: Visualizing Neural Network Training and Adversarial Threats

Many people feel that neural networks are mysterious black boxes; understanding their training process is essential for explaining model biases and protecting AI security, especially as AI merges with traditional information security.

Fundamentally, a neural network is an intelligent simulation of the brain, built from thousands of interacting neurons. Using a convolutional neural network (CNN) as an example, its architecture consists of many layers connected by weighted lines.

The training process simply adjusts all those connection weights to appropriate values.

To help illustrate this, Tencent Zhuque Lab together with Tencent AI Lab released the industry’s first "AI Security Threat Matrix" PDF, which categorises 21 attack types across seven stages of the AI lifecycle, and also produced a series of colorful visualisations of the training process.

Before training, the model looks uniform. During training, the network repeatedly performs the following operations:

Convolution layer : extracts local features from the input image.

Activation layer : applies a non‑linear mapping to the extracted features.

Pooling layer : removes redundant information while retaining key features, preventing the model from being overloaded with unnecessary data.

These three steps are repeated many times, forming a deep network that automatically extracts effective features:

Finally, the fully‑connected layer aggregates all learned features, computes weighted sums, and produces the prediction output.

The article then explores adversarial examples: tiny, almost invisible perturbations added to an input image that cause the model to misclassify. A panda image with added noise is shown to be classified as a car, while the original is correctly identified as a panda.

Feature‑distribution visualisations before and after the adversarial perturbation reveal subtle differences, especially in the third filter of a pooling layer, which act as clues to the model’s drifting prediction.

These minute changes amplify through repeated convolution, activation, and pooling operations—a butterfly effect—ultimately leading to drastically different outputs. Understanding these dynamics demystifies neural networks and underscores the importance of AI security.

For more details, readers are invited to leave comments.