How SCMHSA Improves Transformer Next‑Frame Prediction by Reducing Semantic Dilution

Introduction

Predicting the next video frame is critical for applications such as autonomous driving, object tracking, and motion forecasting. Transformer‑based models have advanced this task but suffer from two major issues: (a) the standard Multi‑Head Self‑Attention (MHSA) splits the input embedding into head‑specific fragments, causing semantic dilution; and (b) the loss is computed on reconstructed pixel frames while the model predicts embeddings, creating a mismatch between training objectives and model outputs.

Proposed Solution

The authors propose a Semantic‑Concentrated Multi‑Head Self‑Attention (SCMHSA) module that processes the full input embedding in each attention head, preserving semantic information. To manage the increased dimensionality, a learnable projection matrix reduces the concatenated head outputs back to the original embedding size while retaining the most relevant semantics.

In addition, a new loss function operates directly in the embedding space. It combines an Embedding MSE Loss that measures the error between predicted and ground‑truth embeddings, with a Semantic Similarity Loss that penalizes heads producing overly similar outputs by computing cosine similarity between head vectors. The total loss is a weighted sum of these two components, with a hyper‑parameter controlling their balance.

Model Architecture

Embedding Layer: A Vision Transformer (ViT) maps each input frame to a low‑dimensional embedding; a learnable [CLS] token aggregates spatial information.

SC‑VFP Encoder: Stacks of Transformer encoder blocks replace the standard MHSA with SCMHSA, allowing each head to attend to the complete embedding.

Prediction Layer: A multi‑layer perceptron (MLP) consumes the encoder output and predicts the embedding of the next frame.

Implementation Details

The model is implemented in PyTorch and trained on an NVIDIA A100 40 GB GPU. It consists of six encoder blocks, each with six attention heads and a 768‑dimensional embedding. The sequence length is five frames. Optimization uses AdamW with a learning rate of 1e‑4, batch size 32, and 25 epochs. Random seeds are fixed to 2023 for reproducibility.

Datasets and Evaluation

Four datasets are used: KTH, UCSD Pedestrian, UCF Sports, and Penn Action. Each training sample contains five input frames and one target frame. Frames are resized to match the ViT input size. Because the method operates in embedding space, traditional pixel‑level metrics (LPIPS, SSIM) are unsuitable; instead, PSNR and MSE are computed on embeddings.

Results

Quantitative comparisons against recent predictors (PredRNN, SA‑ConvLSTM, MIMO‑VP, LFDM, VFP‑ImageEvent, ExtDM) show:

UCSD: Best PSNR = 28.75 dB and MSE = 86.71, improving over the runner‑up by 2.59 % (PSNR) and 16.14 % (MSE).

UCF Sports: MSE reduced by 38.3 % and PSNR increased by 4.84 % relative to ExtDM.

Penn Action: MSE improvement of 68.71 % and PSNR gain of 6.63 %.

KTH: Performance lags (PSNR − 7.01 %, MSE − 59.94 %) due to the dataset’s smaller scale, which lessens semantic dilution.

Qualitative visualizations (Figures 3‑4) illustrate that the predicted embeddings align closely with ground‑truth embeddings, especially on larger datasets.

Ablation Studies

Parameter Analysis: SCMHSA contains 42.7 M parameters versus 31.4 M for the baseline Transformer, reflecting the full‑embedding processing per head.

Performance Analysis: Removing SCMHSA or the Semantic Similarity Loss (SSL) degrades results variably across datasets. For example, on UCSD, excluding SCMHSA raises MSE by 28.87 % and lowers PSNR by 3.97 %; on Penn Action, SSL removal increases MSE by 96.8 % and reduces PSNR by 11.86 %.

Training curves (Figure 5) show faster convergence when SCMHSA is present.

Conclusion

SCMHSA effectively mitigates semantic dilution in Transformer‑based video prediction, and the embedding‑space loss aligns training objectives with model outputs. The combined approach yields substantial accuracy gains, particularly on larger, more complex datasets, confirming its scalability and robustness.

References

[1] Overcoming Semantic Dilution in Transformer‑Based Next Frame Prediction.