How LIMSSR Overcomes Training‑Time Modality Missing for Precise Action Quality Evaluation

Background and Motivation

Real‑world multimodal data are often incomplete because of sensor failures, environmental noise, or privacy constraints. Existing multimodal action‑quality methods typically assume that all modalities are available during training and rely on reconstruction, distillation, or prior fusion to handle missing inputs at test time. This full‑data prior fails when training data themselves contain systematic modality gaps, severely degrading fine‑grained quality scoring.

Proposed Framework: LIMSSR

The authors propose LIMSSR (LLM‑Driven Sequence‑to‑Score Reasoning) to reformulate incomplete multimodal action‑quality assessment as a conditional sequence‑to‑score problem. By leveraging a large language model (LLM) for contextual reasoning, the framework infers the latent semantics of missing modalities from the observable context and then performs cross‑modal fusion and final score regression.

Stage I – Prompt‑Guided Context‑Aware Modality Completion (PCMI)

Instead of zero‑filling missing modalities, LIMSSR inserts explicit placeholder tokens that indicate where and what is missing. For each modality, start‑end boundary tokens are defined; observable modalities are fed with their raw temporal features, while missing modalities receive dedicated missing tokens. The resulting input sequence combines observed features and placeholders, enabling the LLM to predict latent representations for the missing streams.

Stage II – LLM‑Driven Multi‑Dimensional Representation Fusion (LMRF)

LIMSSR introduces multiple fusion tokens that serve as semantic slots for different evaluation dimensions (e.g., motion completeness, temporal stability, rhythm consistency). During inference the LLM integrates multimodal context into these slots, producing a set of token embeddings that are then aggregated by weighted sum to form a primary fused representation.

Stage III – Mask‑Aware Dual‑Path Aggregation (MDA)

To balance semantic inference with statistical robustness, LIMSSR builds two parallel paths. The first path starts from the primary fused representation and, guided by the missing‑modality mask, applies gated residual correction to adjust the confidence of LLM predictions. The second path back‑propagates hidden states of each modality to model cross‑modal statistical relationships, producing a statistical recovery representation whose confidence is also mask‑dependent. A global collaborative coefficient then merges the semantic and statistical paths into the final score prediction, preventing hallucination while preserving rich semantics.

Experimental Results

Table 1 reports LIMSSR’s performance on three public benchmarks (FS1000, Fis‑V, RG) under six incomplete‑modality configurations and the full‑modality baseline. Using Spearman correlation (higher is better) and mean‑squared error (lower is better), LIMSSR consistently achieves the best or highly competitive scores compared with recent methods from incomplete multimodal sentiment recognition, action recognition, and action‑quality assessment.

Table 2 shows that, despite being designed for training‑time missing observations, LIMSSR remains highly competitive when all modalities are present, indicating that the model learns a robust, generalizable multimodal semantic representation rather than sacrificing full‑modality performance.

Analysis of Modality Inference

Figure 3 compares the similarity between inferred modality embeddings and true modality embeddings, demonstrating that inferred representations are closer to their corresponding true modalities than to other modalities, confirming targeted semantic recovery rather than generic averaging.

t‑SNE visualizations (Figure 4) further reveal that the latent representations generated for missing modalities align closely with existing modality clusters, reducing cross‑modal semantic gaps.

Conclusion

LIMSSR addresses the realistic challenge of training‑time incomplete multimodal observations by combining prompt‑guided modality completion, LLM‑driven multi‑dimensional fusion, and mask‑aware dual‑path aggregation. Experiments confirm its superiority under both incomplete and complete settings, highlighting the significant potential of large language models for robust multimodal learning.