Ant Insurance Technology and CASIA Win Two Tracks at MuSe2021 Multimodal Sentiment Challenge (ACM MM 2021)

The 29th ACM International Conference on Multimedia (ACM MM) concluded in Chengdu, drawing over 1,000 scholars from 19 countries and receiving more than 2,000 paper submissions, setting a new record for the conference.

Ant Insurance Technology partnered with the Institute of Automation, Chinese Academy of Sciences, forming a team that won championships in two tracks of the MuSe2021 Multimodal Sentiment Challenge.

MuSe (The Multimodal Sentiment Challenge) is organized by Imperial College London, University of Augsburg, Nanyang Technological University and others, focusing on deep multimodal fusion that incorporates semantic information, and has been held annually during ACM MM since 2020.

In 2021, 32 teams entered four tracks; the Ant‑CASIA team competed in the MuSe‑Wilder and MuSe‑Sent tracks, predicting continuous and discrete labels for arousal and valence. Their solutions achieved the highest average CCC (outperforming second place by 0.1191) and Macro‑F1 (outperforming second place by 0.0671) scores.

The team employed a variety of low‑level handcrafted and high‑level deep learning features from audio, visual, and text modalities, and introduced a self‑attention‑enhanced BiLSTM model that uses CCC loss and F1 loss aligned with the evaluation metrics, markedly improving performance.

For multimodal fusion, they adopted a late‑fusion strategy, adding a second‑level BiLSTM to combine the predictions of each independent modality, which yielded significantly better results than single‑modality predictions.

This competition underscores the interdisciplinary research at the intersection of artificial intelligence and psychology, a current hot topic that attracts substantial investment from leading academic institutions and technology companies.

Ant Group has already deployed multimodal cognition and understanding technologies across insurance, wealth management, security, and customer service, enhancing user experience in scenarios such as screen‑damage insurance and fraud detection, with future plans to apply these methods to critical‑illness claim processing.