How AI Is Revolutionizing Content Safety – The Tech Behind Shanghai’s Top Award

On August 26, the 2024 Shanghai Science and Technology Awards announced that the project "Key Technologies and Applications for Network Media Content Safety Detection in Complex Adversarial Scenarios," jointly completed by Shanghai Jiao Tong University, Ant Group and other institutions, won the First Prize of the Shanghai Science and Technology Progress Award.

The expert committee chaired by Academician Wang Yaonan, a member of the Chinese Academy of Engineering, judged that the overall technology reaches an international advanced level, with AIGC detection and related techniques achieving a leading position worldwide.

With the rise of generative AI and large models, content generation has become highly intelligent but also brings challenges such as value conflicts, ideological issues, and moral violations, threatening healthy information dissemination and the governance of harmful online content. To address this, the collaborating teams proposed key technologies for multimodal understanding, AI verification, anomalous reasoning, and multimodal document analysis in complex adversarial scenarios.

"A sound comprehensive network governance system and a healthy online ecosystem are essential," said Jiang Xinghao, Vice President of Shanghai Jiao Tong University and the project's chief contributor, highlighting the team's effort to protect information content safety in the AI era.

Technology 1: A chain‑of‑thought‑based multimodal large‑model hallucination mitigation technique, developing deep‑learning models that improve generalization and intrinsic safety for content generation, detection, and traceability.

Technology 2: Controllable multimodal data intelligent generation methods, providing anomalous samples needed for model training.

Technology 3: An integrated solution for network media content safety monitoring, tackling the detection of covert harmful content and AIGC material, earning a 5‑star rating from the China Academy of Information and Communications Technology.

Technology 4: An adversarial‑and‑traceability‑based model security protection technique, building an AI model attack‑defense algorithm library to prevent data misuse and model theft.

These innovations significantly boost defense effectiveness and efficiency against complex attacks in the content safety field, delivering breakthrough progress for harmful information governance and providing strong technical support for broader AI applications.

The project has secured 41 patents, contributed one national standard, obtained 10 software copyrights, and published 118 papers. Its technologies have been deployed in multiple platforms, dramatically improving risk‑control response times from days to minutes. AIGC safety detection now powers Alipay’s content services, efficiently identifying and blocking harmful content.

Wang Weiqiang, Chief Scientist of Ant Security Lab and a key project contributor, stated that under the national cybersecurity strategy, Ant Group’s industry‑academia‑research collaboration has built an intelligent governance system for internet information security, offering creators a trustworthy environment and continuously strengthening risk barriers through ongoing AI research.