The article presents UI‑TARS, a native GUI‑agent model that combines multimodal large‑language models with the open‑source Midscene.js framework to enable more accurate, token‑efficient, and privacy‑preserving UI automation, while discussing its architecture, advantages, limitations, and integration steps.
This comprehensive survey examines the foundations, tokenization techniques, model architectures, training paradigms, evaluation benchmarks, and open challenges of multimodal next‑token prediction (MMNTP), offering researchers a clear roadmap for future advances in multimodal AI.
This article details Akulaku’s deployment of large language models across multimodal financial risk‑control scenarios—covering business background, a three‑module intelligent‑agent architecture, concrete tool‑ and planning‑enhancement case studies, and future outlook—demonstrating how LLMs boost efficiency, reduce labeling effort, and enable copilot‑style assistance.
This guide shows how to install the ZhipuAI SDK, obtain a free GLM‑4V‑Flash API key, craft prompts for image‑based calorie estimation, and build a Python demo that calculates food calories, BMI, and personalized diet advice using a multimodal large model.
Ant Group’s AIGC security detection system was evaluated by the China Information and Communication Research Institute, achieving the highest "Excellent" rating with a 0.99 F1 score across image, video, and audio modalities, while also releasing large‑scale detection datasets for the research community.
Midscene.js leverages multimodal AI to simplify web UI automation by providing .ai, .aiQuery and .aiAssert methods, supporting JavaScript and YAML integrations, a Chrome extension, and detailed cost analysis while acknowledging latency, interaction limits, and prompt‑engineering challenges.
The article introduces two 360 AI Research Institute projects—IAA, an architecture that equips frozen language models with multimodal capabilities via plug‑in layers, and BDM, a Chinese‑native diffusion model compatible with the Stable Diffusion ecosystem—detailing their motivations, designs, benchmark results, and open‑source resources.
Midscene.js, an open‑source UI automation framework from ByteDance Web Infra, leverages multimodal AI to simplify writing, maintaining, and debugging web UI tests with JavaScript or YAML integrations, while discussing its origins, usage patterns, limitations, cost, and security considerations.
This article explains why GUI agents are needed, defines their multimodal capabilities, walks through a high‑level automation scenario, details the architecture of large‑model‑driven GUI agents, highlights recent open‑source projects, and compares them with traditional RPA solutions.
At the 2024 Global Machine Learning Conference in Beijing, 360 AI Research Institute showcased cutting‑edge multimodal large‑model research, fine‑grained open‑world object detection, and document understanding technologies, highlighting open‑source releases, real‑world deployments, and competitive achievements in AI competitions.
iQIYI’s multimodal speaker diarization system splits long‑form video using subtitle timestamps and scene detection, extracts voiceprints with a custom model, hierarchically clusters them, and applies an Activate Speaker Detection algorithm combined with face‑recognition to assign speakers, achieving around 90 % precision and recall and boosting downstream tasks such as summarization, translation, and dubbing.
This article presents the research behind Synchronized Video Storytelling, introducing the E‑SyncVidStory dataset, the VideoNarrator multimodal architecture, and extensive experiments that demonstrate high‑quality, product‑aware video narration generation for e‑commerce applications.
This article introduces embodied intelligence, explains its basic definition, three essential elements (body, intelligence, environment), and details the four functional modules—perception, decision, action, and feedback—while describing the sensors and algorithms that enable physical AI systems to interact with the real world.
By leveraging multimodal large models such as GPT‑4o and fine‑tuned Qwen‑VL, the study builds a prompt‑engineered and SFT‑enhanced voucher verification system that classifies product categories, detects diverse defects, and estimates problem counts, achieving up to 90 % accuracy and meeting real‑time business throughput requirements.
This survey acts as a comprehensive portal that organizes AIGC research across seven domains—text, image, and audio generation, cross‑modal association, text‑guided image and audio synthesis, and supporting resources—detailing seminal models such as GPT, Diffusion, CLIP, DALL·E, Stable Diffusion, MusicLM, and key papers that shaped each field.
ChatGPT 4.0 with Canvas introduces a visual "canvas" that blends language and graphics, enabling multimodal dialogue, real‑time visual feedback, and collaborative workflows across education, design, and business, while posing technical challenges in vision‑language integration, context consistency, and performance optimization.
The article describes a multimodal AI video moderation system built on Alibaba’s Chinese‑CLIP model and hybrid RedisSearch/ElasticSearch vector databases, enabling real‑time violation detection and historical recall, with fine‑tuned black‑market ad detection, FP16 quantization, and OpenVINO acceleration to boost speed and cut storage.
Mistral AI, a French startup founded in 2023, leverages open‑source large language models, efficient architecture, and multimodal research to offer scalable AI solutions across enterprises, content creation, and healthcare, while pursuing a community‑driven strategy that positions it as a rising force in the competitive AI landscape.
Huolala, a logistics technology company, shares how it leverages large language models to create personal and office AI assistants across dozens of real‑world scenarios, detailing the underlying platform, prompt engineering, multimodal capabilities, multi‑agent coordination, and the resulting business empowerment.
The article examines how AI‑enhanced search, driven by large language models, is reshaping the competitive landscape, highlights Baidu’s multi‑modal features and market positioning, and explains why intelligent search is emerging as a high‑potential, user‑centric product in a crowded industry.
The article presents a comprehensive overview of Kuaishou’s Kolors (formerly 可图) multimodal generative model, detailing its data collection strategy, diffusion‑based architecture, evaluation metrics, derived capabilities such as prompt refinement and interactive generation, and a range of practical applications from AI‑powered live‑stream gifts to virtual try‑on, while also offering strategic advice for the domestic visual‑generation community.
This article introduces Alibaba Tongyi Lab's multimodal mobile AI agent, Mobile‑Agent, covering the background of large‑model agents, the design and capabilities of V1 and V2, the multi‑agent framework, evaluation results, open‑source resources, and future development directions.
This article compares Baidu’s new AI writing platform Orange Paper with Kimi, evaluating their long‑text understanding, multimodal editing, document upload limits, outline generation, and overall usability for research and academic writing, highlighting Orange Paper’s advantages in knowledge retrieval, large‑scale content creation, and deep editing capabilities.
This article reviews Tencent Music Tianqin Lab’s research on audio representation large models, covering background, the evolution of audio features, self‑supervised methods such as SimCLR, BYOL, MAE, MLM, benchmark results, multimodal extensions, and real‑world applications like song authenticity detection and search ranking.
The article details Kuaishou’s development of the 175B “Kuaiyi” multimodal large model, presenting eight novel technical innovations—from Temporal Scaling Law and MiLe Loss to MoE‑enhanced reward modeling—and describes how these advances enable high‑performance AI services such as the AI Xiao Kuai chatbot across diverse real‑world scenarios.
This article summarizes the development background of large language models, Alibaba's progression in foundational and personalized AI, the design and capabilities of the Tongyi Xingchen personalized model, its multimodal and agent-based architecture, various industry use cases, and the safety and responsibility measures applied to ensure trustworthy AI deployment.
In a candid interview, Geoff Hinton reflects on his AI journey—from early disappointments in physiology and philosophy to breakthroughs in neural networks, scaling laws, multimodal learning, fast‑weight concepts, and the ethical challenges shaping the future of artificial intelligence.
This article presents Ant Group's multimodal research on video retrieval, detailing a large Chinese video‑text pre‑training dataset, three techniques that raise video‑text semantic search performance by up to 24.5%, and an end‑to‑end video‑video copyright detection system that reduces storage by 85% and speeds up inference 18‑fold.
xAI’s Grok 1.5V is set to support multimodal image input, allowing developers to upload pictures and receive text‑based answers via the Python SDK, marking a major upgrade that narrows the gap with leading models like GPT‑4 and signals a new frontier for AI chatbots.
Google’s new ScreenAI visual‑language model, built on the PaLI architecture, can interpret user interfaces and infographics, answer UI‑related questions, generate summaries and navigate screens, and sets new benchmarks that may reshape future user‑experience research and applications.
This article presents Ant Group's multimodal research on video retrieval, detailing video‑text semantic search and video‑video same‑source search, introducing a large Chinese pre‑training dataset, novel pre‑training, hard‑sample mining, fine‑grained modeling techniques, and an efficient end‑to‑end copyright detection framework.
OpenAI introduced GPT‑4o, a free, omni‑capable multimodal model that processes text, audio, and images together, delivers near‑human response latency, showcases impressive live demos, and will soon be available via a discounted API, marking a significant step forward in end‑to‑end AI research.
OpenAI’s latest flagship model GPT‑4o combines text, audio, image and video processing in a single, faster, cheaper multimodal system that delivers near‑human response times, expanded API access, and new safety measures, reshaping how developers and users interact with AI.
This article presents JD Advertising's 2023 AI-driven framework that uses a relation‑aware diffusion model with visual‑text and geometric modules, combined with category‑common and personalized generators and a planning‑and‑rendering network, to automate high‑quality, scalable e‑commerce poster creation and background synthesis.
Sora, a new text‑to‑video AI model, can create one‑minute videos from textual prompts or static images, delivering industry‑leading fidelity, resolution, and coherent motion by using spatial‑temporal patches inspired by ViViT, and shows emergent capabilities that hint at universal physical simulation.
A research team from Sun Yat‑sen University, Sea AI Lab and Harvard built a multimodal large model that learns to generate creative jokes and memes by training on the Oogiri‑GO dataset, introducing a Leap‑of‑Thought (LoT) paradigm and CLoT fine‑tuning, which outperforms GPT‑4 and other state‑of‑the‑art models in humor tasks.
The article reviews 2023's rapid LLM advances, public hype versus long‑term reality, the lack of hard limits to AGI, the rise of imagination‑driven capabilities, startup challenges, model compression, multimodal breakthroughs, AI agents, and the persistent US‑China technology gap.
In a detailed interview, Tsinghua researcher Zhao Sicheng and JD Retail senior director Peng Changping discuss the latest progress in multimodal large models, their practical applications in advertising and e‑commerce, persistent challenges such as hallucinations and data alignment, and the skills engineers need to thrive in the emerging AI era.
Alibaba Cloud’s AI platform PAI showcased four cutting‑edge papers at AAAI‑2024—introducing AMFormer for deep tabular learning via arithmetic feature interaction, MuLTI for efficient video‑language understanding, M2SD for few‑shot class‑incremental learning, and M2Doc for multi‑modal document layout analysis—demonstrating the platform’s growing impact on artificial‑intelligence research.
The article analyses the 2024 AI landscape, highlighting a multimodal explosion, the limits of current AI applications, Sora as a concrete step toward AGI, the rise of AI‑native business models, edge‑AI hardware opportunities, the challenges of human‑level models, and the broader societal impacts of an AI‑driven data era.
This comprehensive review examines Sora, the first model capable of generating minute‑long, high‑quality videos from text, covering its historical background, core diffusion‑Transformer architecture, data preprocessing strategies, prompt engineering techniques, diverse applications, and the ethical and technical limitations that shape its future.
Anthropic has launched the Claude 3 family of large language models—Opus, Sonnet, and Haiku—offering varying balances of intelligence, speed, and cost, with enhanced reasoning, multilingual, vision capabilities, reduced refusals, and improved safety, now available via API in over 159 countries.
This article presents a comprehensive overview of document intelligence, covering its background, technical evolution, large‑model advancements, and practical enterprise digital transformation use cases, with a focus on multimodal processing, unified document representation, and industry‑specific applications such as legal contract automation.
document intelligenceenterprise automationlarge language models
Claude 3 has arrived, outperforming GPT‑4 across benchmark scores, offering free Sonnet and paid Opus tiers, and showcasing unprecedented multimodal, long‑context, and code‑generation abilities that reshape competitive dynamics in large‑language‑model research.
Ant Group introduced SkySense, a 2.06‑billion‑parameter multimodal remote‑sensing foundation model that outperformed 18 international rivals across 17 benchmark tasks, was accepted to CVPR 2024, and aims to support applications such as agriculture, urban planning, and disaster response.
The article provides an in‑depth technical analysis of OpenAI's Sora, highlighting its 60‑second 1080p video generation capability, the novel patches‑vectorization and transformer training pipeline that leverages GPT‑generated prompts for multimodal alignment, and its potential to become a universal video‑generation base model that could reshape the AI industry.
The article analyzes OpenAI's Sora video model, arguing that its integration of large‑language‑model reasoning with diffusion techniques marks a major step toward true world understanding, reshapes creative workflows, widens the AI talent gap, and accelerates the path to artificial general intelligence.
The Mobile-Agent paper presents a vision‑only, autonomous multi‑modal AI system that can interpret user commands, locate UI elements on a smartphone screen, and execute complex tasks such as browsing, commenting, and content creation through a defined operation space, self‑planning, and self‑reflection mechanisms, achieving high success rates across diverse Chinese and English scenarios.
This article introduces LLaVA, an open‑source large language‑visual assistant that replicates GPT‑4‑V capabilities, explains its architecture, training process, and key features, and provides step‑by‑step instructions for using the web demo, running it locally via Ollama or HuggingFace, and building a simple Gradio chatbot with code examples.
The article presents a comprehensive overview of cross‑modal video open‑tag mining, detailing its technical background, related multimodal research methods, a four‑stage open‑tag solution from 360 AI Research Institute, and future application prospects such as unsupervised tag coverage, semantic retrieval, and content moderation.
Xiaohongshu’s recommendation system uses large‑scale multimodal embeddings, dual‑tower and graph models, and diversity techniques like DPP and SSD to quickly surface high‑quality user‑generated content, enabling ordinary users to gain visibility while balancing personalization, exploration, and efficient LLM‑augmented pipelines.
This article reviews the evolution of multimodal large models, introduces Jiuzhang Yunji DataCanvas' multimodal model platform—including AI foundation software, model tools, serving, and prompt management—shares practical building methods, memory‑augmented models, ETL pipelines, knowledge‑base applications, and offers a forward‑looking perspective on enterprise data management and intelligent agents.
This article reviews the latest research from Alibaba DAMO Academy on real-world image quality problems, covering spatial, temporal, and color enhancement methods, advanced quality assessment metrics, multimodal diffusion models, and future directions toward large‑model integration and lightweight deployment.
The article examines Google’s Gemini multimodal AI demo, revealing that the striking video was largely fabricated using static image frames and engineered prompts, which misleads viewers about the model’s real‑time capabilities and raises concerns about trust in AI demonstrations.
Google's Gemini AI suite, unveiled in December, brings three model sizes—Nano, Pro, and Ultra—to power Bard and other services, claims superior performance over GPT‑4 across most benchmarks, and introduces multimodal capabilities that signal a major shift in the AI landscape.
Google’s Gemini, a trillion‑parameter native multimodal model trained on TPU v4/v5e, was launched overnight and, according to its technical report, surpasses GPT‑4 on 30 of 32 academic benchmarks, achieves the first human‑level score on MMLU, and powers the new AlphaCode 2 code‑generation system.
This article presents Nine Chapters Cloud Extreme DataCanvas's insights and practices on multimodal large model platforms, covering their historical development, platform components such as AI Foundation Software and Prompt Manager, practical implementations like memory-augmented models and ETL pipelines, and future prospects for enterprise knowledge bases and agents.
The authors present a multimodal system that automatically extracts high‑energy video segments for dynamic covers by analyzing subtitles, audio, visual frames, and danmu, employing LLM prompt‑tuning, scene‑cut detection, and aesthetic scoring to reduce manual effort and boost click‑through rates.
OpenAI’s latest update adds multimodal voice and image capabilities to ChatGPT, letting users speak or upload pictures for more natural, context‑rich conversations powered by advanced GPT‑3.5 and GPT‑4 models.
This article presents a comprehensive overview of document intelligence, covering its background, the evolution of related technologies, large‑model approaches such as multimodal pre‑training and domain‑specific models, and concrete enterprise use cases across various business functions.
Document Understandingdocument intelligenceenterprise AI
This article introduces Kuaipedia, Kuaishou's multimodal short‑video encyclopedia, detailing its background, system architecture, knowledge‑video recognition pipeline, multimodal entity linking techniques, and downstream applications, while also providing implementation insights and a brief Q&A.
mPLUG-Owl is a modular multimodal dialogue large model from Alibaba DAMO Academy that builds on the mPLUG series, offering advanced image, video, OCR, and multilingual capabilities, with extensive evaluations showing superior performance over MiniGPT‑4, LLaVA, and other multimodal LLMs across various tasks.
Baidu’s chief architect Li Shuanglong outlined how AIGC, driven by advanced large‑language and multimodal models, is already powering commercial tools such as automated copywriting, 2D digital‑human video creation and lead‑generation chatbots, while emphasizing future progress in engineering scalability, algorithmic fidelity, data quality, and scenario‑focused applications.
Three papers from Alibaba Cloud's PAI platform were selected for ACL 2023, showcasing FashionKLIP for e‑commerce image‑text retrieval, ConaCLIP's lightweight dual‑encoder distillation, and a fast domain‑specific diffusion model, all of which will be open‑sourced for the AI community.
OpenAI has opened GPT‑4’s API to all paid users, offering an 8K‑token context window (up to 32K), multimodal image input, enhanced creativity, longer text handling, and upcoming fine‑tuning options, while also outlining phased deprecation of older models and current limitations.
In this expert roundtable, leading AI researchers discuss GPT‑4’s multimodal breakthroughs, the future convergence of large models with knowledge graphs, practical integration strategies, and the evolving relevance of traditional NLP tasks, offering deep insights into the direction of artificial intelligence research.
This article presents Ant Security's research on novel biometric methods—brainwave (脑纹) and behavior recognition—detailing their scientific background, data collection, multi‑modal deep‑learning algorithms, adversarial and contrastive training strategies, experimental results, and practical applications for inclusive, secure identity verification.
In this interview, Jina AI’s founder Shao Han examines the relationships among generative, perceptual, and decision AI, compares single‑modal and multimodal approaches, discusses large language model development, and evaluates the impact of ChatGPT on search and future AI commercialization.
In a SETalk livestream, experts discuss how ChatGPT and Copilot X dramatically boost coding efficiency, lower entry barriers, reshape software development workflows, raise concerns about job displacement, and outline three emerging paradigms—LLM for SE, SE for LLM, and LLM as SE—while highlighting current limitations and future trends.
GPT‑4, the new multimodal AI model, can process images and text, generate code and natural language, achieve human‑level scores on standardized exams, handle up to 32 K tokens, and demonstrates advanced reasoning, while OpenAI emphasizes its safety improvements and current limitations as a still‑emerging technology.
A recent 154‑page Microsoft paper titled "Sparks of Artificial General Intelligence: Early Experiments with GPT‑4" argues that GPT‑4, despite being an early prototype, already exhibits many capabilities—multimodal reasoning, programming, mathematics, and human‑like interaction—suggesting it may be an early form of AGI, though experts highlight significant limitations and ongoing debates.
Baidu Netdisk’s new image search combines ERNIE‑ViL‑based semantic vectors, cross‑modal matching and metadata such as timestamps, GPS and facial tags, using LSH‑optimized indexing to let users find specific photos among billions with natural‑language queries, delivering faster, more accurate results without manual tagging.
Microsoft's open‑source Visual ChatGPT extends ChatGPT with image send/receive capabilities, explains its multimodal architecture, demo scenarios, used visual models, and points to the arXiv paper, highlighting its rapid popularity growth on GitHub.
NetEase’s Fuxi team developed the multimodal ‘YuZhi’ model, a large‑scale image‑text dual‑tower system optimized with the EET inference framework, which powers personalized recommendations in NetEase News and Cloud Music, while a partnership with Huawei Ascend AI and MindSpore enables further model acceleration, compression, and the new ‘YuZhi‑Wukong’ model that improves video recommendation metrics by about 5%.
The article presents a side‑by‑side demonstration of Baidu’s newly released Wenxin Yiyan and OpenAI’s GPT‑4 across literary creation, business copywriting, mathematical reasoning, Chinese idiom interpretation, acrostic poetry, and multimodal generation, then explains the underlying six‑core technologies and Baidu’s hardware‑cloud strategy while reporting audience reactions.
This article provides a detailed analysis of GPT‑4, covering its multimodal abilities, performance gains, training innovations, safety improvements, new application scenarios, impact on developers, and future trends in large language models.
OpenAI’s GPT‑4, a multimodal large language model that accepts text and image inputs, powers ChatGPT and Bing, offers improved creativity and problem‑solving while still facing hallucination risks, and is now available via ChatGPT Plus and an open API for developers.
Microsoft Germany's CTO announced the imminent release of a multimodal GPT‑4, highlighting its ability to process text, images and video, while executives liken the breakthrough to an “iPhone moment” for AI, emphasizing new capabilities, industry disruption, and responsible data use.
Alibaba’s internal‑test “Damoyuan‑version ChatGPT” showcases a multimodal AI that combines text, image, code, and creative generation, built on the Tongyi large model’s unified architecture, while other Chinese tech giants rush to launch their own ChatGPT‑style products.
The article reviews the evolution of large language models, highlighting two major paradigm shifts after GPT‑3, the role of scaling laws, knowledge acquisition, prompting techniques, reasoning abilities, and outlines future research priorities for building more capable and efficient AI systems.
On January 17, the China Academy of Information and Communications Technology announced that Ant Security's self‑developed Tianjian multimodal content risk control system achieved the highest five‑star rating in both text and image assessments of the 2022 content review service evaluation, highlighting its advanced AI‑driven moderation capabilities.
This article introduces the rapid development of multimodal digital humans powered by large AI models, covering their background, current challenges, NeRF‑GAN based modeling methods, multimodal dialogue capabilities, and real‑world application cases such as virtual assistants, tourism guides, and sign‑language avatars.
In a candid conversation, Sam Altman and Reid Hoffman explore the next stage of AI, discussing commercial opportunities of large language models, the rise of AI‑plus applications in science and the metaverse, future directions such as multimodal and continuously learning models, and the societal challenges of AGI, wealth distribution and universal basic income.
In this interview, NetEase AI expert Li Yuke shares how lightweight, cost‑effective AI solutions improve digital content risk control, audio‑video processing, and conversational systems, while discussing technical committees, data standards, and future AI trends such as multimodal and unsupervised learning.
This article provides a comprehensive overview of Volcano Engine's virtual digital human platform, detailing its definition, AI‑driven and human‑driven classifications, 2D and 3D technical architectures, multi‑modal perception, interaction capabilities, application scenarios, and future development directions.
This article introduces the evolution of digital human technology, explains the five maturity levels (L1‑L5), describes the Motionverse multimodal motion‑generation platform and its large‑scale data and AI models, and outlines SDK integration strategies for diverse metaverse scenarios.
At the REDtech Youth AI Technology Salon in Beijing, leading AI experts and top university students discussed the evolution of multimodal learning, Xiaohongshu’s practical applications, autonomous‑driving perception, and offered career guidance, emphasizing solid fundamentals, user value, and opportunities within Xiaohongshu’s talent‑development programs.
Liu Wei outlines Tencent’s Advertising Multimedia AI ecosystem on the Taiji platform, describing a five‑platform matrix—Jue for content understanding, Qiankun for automated video creation, Shenzhen for AI‑driven review, Tianyin for hierarchical fingerprinting, and Hunyuan as a multimodal large model—featuring innovations such as massive multimodal pre‑training, logo retrieval, QA‑style attribute extraction, spatiotemporal video analysis, advanced auto‑judgment, and high‑performance hashing that achieve top cross‑modal retrieval results.
Xiaohongshu’s in‑house LarC platform powers real‑time, multimodal recommendation, life‑search, and generative‑AI commercial content for its 200 million‑user community by processing billions of daily feedback samples, employing conflict‑free parameter servers, diversified sequence modeling, and large‑scale representation learning to deliver personalized, fresh, and diverse user experiences.
AI infrastructureMachine Learning PlatformRecommendation System
The article introduces the GIT image‑to‑text (image captioning) model, explains its transformer‑based architecture, showcases multiple example outputs, discusses training details, compares its performance with Flamingo and COCO, and highlights its applicability to tasks such as VQA, video captioning, and image classification.
This article introduces the importance of image‑text cross‑modal representation, presents the Chinese Zero dataset with two pre‑training subsets and five downstream tasks, describes the R2D2 dual‑tower‑plus‑single‑tower pre‑training framework with multiple loss functions, and reports extensive experiments and real‑world deployment insights.
Alibaba’s advertising AI team presents a comprehensive study of four new multimodal copywriting tasks—image overlay text generation, video narration, text style transfer, and detail-page extraction—detailing model architectures, training on billions of images, experimental results, and practical deployment in the “Xiyu” product.
advertising copydetail extractionlarge-scale training
This article reviews Alibaba's digital‑human (virtual avatar) research over the past few years, covering the product’s evolution, a six‑stage pipeline for building digital humans, solutions to key challenges in realism, multimodal interaction, and the open‑source MMTK algorithm library.
This article reviews Alibaba's XiaoMi digital human technology, covering its evolution since 2019, a six‑stage pipeline for building avatars, methods to enhance emotional, textual, vocal, and motion expressiveness, and approaches for improving long‑term interactive capabilities such as controllable script generation, multimodal QA, sign‑language translation, and intelligent behavior decision, culminating in the release of the MMTK multimodal algorithm library.
The framework tags short videos by fusing text, image and audio‑video features through specialized extraction, classification, generative and retrieval modules, then ranking candidates with a multimodal BERT model, delivering accurate, business‑specific tags that boost recommendation, search and advertising.
The article reviews the 2021 AI landscape, highlighting the race for ever‑larger pretrained models, the dominance of Transformers across modalities, the promise and limits of large models, the rise of multimodal systems, regulatory considerations, and the still‑nascent progress in reinforcement learning.
Taobao’s Double 11 showcased a suite of advanced technologies—including 3D immersive XR live rooms, large‑scale multimodal product search, ultra‑low‑latency streaming, AI‑driven content moderation, and low‑carbon model compression—demonstrating how e‑commerce can innovate while promoting sustainability and inclusivity.
This article reviews ByteDance’s intelligent audio signal processing technologies, covering foundational algorithms, multimodal audio scaling, sound‑field reconstruction, and high‑quality low‑latency VoIP, and explains how these advances improve audio capture, immersive media, and smart voice interaction across devices.
This talk explores how multimodal AI, knowledge‑graph‑enhanced script generation, and advanced reading‑comprehension techniques enable virtual anchors to transform e‑commerce live streaming from simple Q&A bots into interactive, content‑rich live broadcasts, addressing challenges of material sourcing, personalization, and low‑latency response.
This article introduces a multimodal advertisement detection framework for WeChat KanKan that decomposes the problem into text, image, and article‑structure dimensions, presents novel models for ad text and image recognition, and describes how sequence classification and visualisation are used to filter severe ad‑spam articles.