SIF3D: Sense‑Informed Forecasting of 3D Human Motion with Multimodal Attention

Imagine a smart home that predicts your movement and opens a cabinet door before you reach it. This scenario illustrates the potential of SIF3D (Sense‑Informed Forecasting of 3D human motion), a scene‑aware human motion prediction framework recently accepted at CVPR 2024.

SIF3D leverages three modalities—observed motion sequences, 3D scene point clouds, and human gaze—to predict future human trajectories and poses in complex environments. The core of the method consists of two novel attention mechanisms:

ternary intention‑aware attention (TIA) : aggregates motion features, extracts global salient point clouds from the scene, and incorporates gaze information to infer the person’s intent and guide trajectory prediction.

semantic coherence‑aware attention (SCA) : operates frame‑wise to identify local salient point clouds that are semantically consistent with each pose, thus assisting pose prediction.

The processing pipeline includes:

Encoding : a PointNet++ network encodes the 3D scene, while a Transformer encodes the motion sequence. Gaze points are indexed into the scene features.

Cross‑modal attention : TIA extracts global salient points, SCA extracts local salient points, and both sets of features are fused with the motion encoder output.

Decoding : a graph‑convolutional decoder merges the trajectory and pose predictions, and a discriminator further refines the realism of the generated motion.

Extensive experiments on the GIMO and GTA‑1M datasets show that SIF3D achieves state‑of‑the‑art performance on both trajectory (Traj‑path, Traj‑dest) and pose (MPJPE‑path, MPJPE‑dest) metrics, outperforming recent graph‑based (LTD, SPGSN) and transformer‑based (AuxFormer) baselines as well as the scene‑aware BiFu method.

Ablation studies confirm the importance of each component (TIA, SCA, point‑cloud encoder, decoder, discriminator) and reveal that using the last frame’s motion feature for TIA yields the best results. The method also demonstrates a good trade‑off between point‑cloud size (4096 points) and computational cost.

Overall, SIF3D introduces a pioneering multimodal framework that tightly couples 3D scene perception with human intent modeling, pushing forward the frontier of human motion forecasting in realistic environments.