Hierarchical Graph Convolutional Networks for Video Social Relationship Modeling

The rapid growth of online social media platforms has created a strong demand for fine‑grained video retrieval and semantic summarization services, yet existing video understanding methods lack deep semantic cues and often ignore the social relationships among characters.

To address this gap, the authors propose a hierarchical graph convolutional network that integrates short‑term visual, textual, and auditory signals and aggregates them through two levels of graph convolutions to produce a global social relationship graph.

Problem Background : Traditional video understanding focuses on describing actions and identities without capturing deeper semantics such as why characters exhibit different emotions, which limits accurate storyline comprehension.

Related Work : Early image‑based social relationship recognition (e.g., PIPA, PISC) evolved to video datasets like MovieGraphs and ViSR. The CVPR 2019 MSTR framework introduced intra‑, inter‑, and triple‑graphs but still conflated interaction behavior with relationship semantics.

Proposed Enhancements : By incorporating multimodal textual information (dialogue, live comments) and designing a weakly supervised loss, the new model can infer relationships even when characters do not appear simultaneously.

Relationship Graph Generation consists of three modules:

Frame‑level Graph Convolutional Network: builds sub‑graphs for each frame using person nodes (C), pair nodes (P), background nodes (G), and temporal text nodes (T).

Multi‑channel Temporal Accumulation: employs two LSTMs to capture dynamics of individual appearances (C) and pairwise interactions (P).

Segment‑level Graph Convolutional Network: merges frame‑level sub‑graphs into richer segment‑level graphs, adding audio dialogue features.

The model is trained with weak supervision, using only segment‑level relationship labels to avoid costly frame‑wise annotation.

Experimental Results : Evaluations on the public ViSR dataset and a self‑built Bilibili dataset (which includes bullet‑screen comments) show notable performance gains, especially for hostile relationships and scenarios with many characters.

Applications :

Social‑relationship graphs improve user experience by providing clearer plot explanations and supporting semantic applications such as storyline description and causal linking.

Social‑aware video person retrieval leverages relationship priors to filter candidates, outperforming pure visual re‑identification methods, particularly under occlusion or extreme pose variations.

Future Outlook : The authors envision further integration of multimodal cues with graph‑based representations to achieve dynamic, semantic‑rich video understanding, extending beyond static visual relations to richer scene graphs.