This article provides a comprehensive, step‑by‑step analysis of Diffusion Policy for robot visuomotor control, covering its motivation, task characteristics, model design, dataset preparation, training pipeline, inference procedure, experimental results, and open research questions.
PixelHacker introduces a latent class guidance (LCG) paradigm that injects foreground and background embeddings into a diffusion model, training on 14 million image‑mask pairs and achieving state‑of‑the‑art structural and semantic consistency across Places2, CelebA‑HQ and FFHQ benchmarks.
This comprehensive survey reviews the rapid progress of multimodal understanding and text‑to‑image generation models, categorises existing unified architectures into diffusion‑based, autoregressive, and hybrid paradigms, analyses their tokenisation strategies, datasets and benchmarks, and highlights current challenges and future research directions.
DP-Recon leverages generative diffusion priors and a visibility‑guided SDS loss to achieve high‑fidelity, compositional 3D scene reconstruction from extremely sparse images, delivering superior geometry, texture, and text‑driven editing capabilities demonstrated on benchmark datasets and real‑world indoor scenarios.
DiffDenoise introduces a three‑stage self‑supervised pipeline that combines a blind‑spot network, conditional diffusion modeling, and stabilized reverse diffusion sampling to dramatically improve medical image denoising performance on both synthetic and real datasets, while also offering a fast distilled version for practical deployment.
PosterMaker leverages a ControlNet‑based TextRenderNet with character‑level visual features and a reward‑driven foreground‑extension detector to generate high‑quality product posters that accurately render Chinese text (over 90% sentence accuracy) while preserving product fidelity, and is already deployed in Alibaba’s AI creative tool.
FloED introduces a dual‑branch, flow‑guided diffusion framework that dramatically improves spatio‑temporal consistency and computational efficiency for video restoration, outperforming existing text‑guided diffusion methods on both object removal and background repair benchmarks.
FlexIP introduces a dual‑adapter architecture and a dynamic weight‑gating mechanism that decouple identity preservation from personalized editing, allowing continuous control over image generation and outperforming prior SOTA methods in both fidelity and flexibility.
DCEdit introduces a precise semantic localization strategy and a dual-level control mechanism for text‑guided image editing, delivering superior background preservation and editing quality, as demonstrated on the new RW‑800 benchmark and extensive comparisons with state‑of‑the‑art diffusion models.
TurboFill introduces a fast image‑inpainting model that trains a repair adapter on a few‑step text‑to‑image diffusion backbone, achieving state‑of‑the‑art results with only four diffusion steps while dramatically reducing computational cost.
Flow Matching reformulates generative modeling by learning a time‑dependent vector field that deterministically transports Gaussian noise to data, using a neural network trained with an analytically derived L2 loss, yielding simpler training, faster convergence, and deterministic sampling that matches or exceeds diffusion model quality.
MagicColor introduces a novel multi‑instance sketch‑coloring framework that uses a two‑stage self‑play training strategy, instance guidance, and edge‑aware pixel‑level color matching to automatically produce high‑quality, consistent colors for multiple line‑art instances, outperforming prior GAN and diffusion‑based methods.
The article explains how knowledge distillation transfers capabilities from large to smaller diffusion models, covering hard and soft labels, temperature scaling, and contrasting it with data distillation, while detailing techniques such as consistency models, progressive distillation, adversarial distillation, and adversarial post‑training for model compression and step reduction.
OmniPaint introduces a unified diffusion‑based framework that achieves physically consistent object removal and insertion by leveraging a pre‑trained FLUX‑1 diffusion prior, a progressive CycleFlow training pipeline, and a novel reference‑free CFD metric for high‑fidelity image editing.
ImageRAG introduces a retrieval‑augmented generation framework that dynamically fetches relevant images to guide diffusion models, dramatically improving the synthesis of rare and fine‑grained concepts across multiple text‑to‑image systems, as demonstrated by extensive quantitative and user studies.
The article analyzes Mercury Coder, a diffusion‑based language model that generates text and code in parallel, compares its speed and quality against traditional autoregressive LLMs like GPT‑4o‑mini using a ball‑collision benchmark, and discusses the underlying score‑entropy training, current limitations, and future multimodal potential.
VideoPainter introduces a plug‑and‑play dual‑branch framework for video inpainting and editing, featuring a lightweight context encoder, ID‑consistent resampling, and the large VPData/VPBench datasets, and achieves state‑of‑the‑art results across eight quantitative and qualitative metrics.
This article explains how recent MIT research uses stochastic differential equations to model diffusion and flow processes, defines training objectives, explores conditional guidance, compares U‑Net and diffusion transformers, addresses memory challenges with latent diffusion, and surveys applications ranging from robotics to protein design.
IMAGPose introduces a unified conditional diffusion framework that combines feature‑level, image‑level, and cross‑view attention modules to generate high‑fidelity, photo‑realistic person images under diverse pose and multi‑view scenarios, outperforming prior SOTA methods on DeepFashion and Market‑1501.
The paper introduces a novel attention‑distillation loss and a guided‑sampling scheme that together enable diffusion models to faithfully transfer visual features from reference images, dramatically speeding synthesis and surpassing prior plug‑and‑play attention methods across style transfer, text‑to‑image generation, and texture synthesis tasks.
This paper introduces a lightweight Diffusion Restoration Adapter that integrates into pre‑trained diffusion priors such as StableDiffusion XL and StableDiffusion 3, dramatically reduces parameter overhead compared with ControNet, and delivers superior quantitative and visual results on real‑world image restoration benchmarks through a novel sampling strategy.
JD Retail’s self‑built AIGC platform ‘JingDianDian’ leverages multimodal diffusion models, ControlNet, RAG and reinforcement learning to automatically generate high‑quality product images, videos and marketing copy, cutting production time from days to seconds, slashing costs by over 99% for more than 350 k merchants.
DC‑AE introduces Residual Autoencoding and Decoupled High‑Resolution Adaptation to achieve up to 128× spatial compression in autoencoders, preserving reconstruction quality while delivering roughly 19× inference and 18× training speedups for high‑resolution diffusion models, as demonstrated on ImageNet and other benchmarks.
DC-ControlNet introduces intra‑ and inter‑element controllers that decouple global conditions into separate content and layout signals, enabling finer‑grained, conflict‑aware control of multi‑condition image generation and achieving higher flexibility and accuracy than traditional ControlNet approaches.
The TPAMI 2025 paper by researchers from Peking University, KAUST, and ByteDance introduces Invertible Diffusion Models (IDM), an end‑to‑end trainable, memory‑efficient diffusion framework that narrows the gap between noise estimation and image reconstruction, reduces sampling steps from 100 to 3, boosts PSNR by 2 dB, and speeds inference up to 15×, with open‑source code available.
ImageRAG introduces a dynamic retrieval‑augmented generation framework that integrates visual language models and CLIP‑based similarity search to supply reference images, enabling diffusion models like OmniGen and SDXL to better render rare and fine‑grained concepts, as demonstrated through extensive quantitative and qualitative experiments.
FlashVideo introduces a two‑stage diffusion framework that first ensures prompt fidelity at low resolution with a 5‑billion‑parameter DiT, then efficiently adds fine details at high resolution using flow matching, achieving state‑of‑the‑art quality with dramatically lower compute cost.
SnapGen is a 379 M‑parameter text‑to‑image diffusion model that produces 1024 px images on mobile devices in about 1.4 seconds, using a compact U‑Net design, multi‑stage knowledge distillation, step distillation, and optimized training tricks to outperform much larger models on standard benchmarks.
The paper introduces DiC, a fully convolutional diffusion model that rethinks 3×3 convolutions, adds sparse skip connections, stage‑specific embeddings and conditional gating, and demonstrates superior FID/IS scores and faster inference compared to diffusion Transformers across multiple scales.
ArtCrafter introduces a novel text‑image style transfer framework that leverages attention‑based style extraction, text‑image alignment enhancement, and explicit modulation to achieve controllable, diverse, and high‑fidelity visual results, outperforming existing methods in both qualitative and quantitative evaluations.
SnapGen introduces a compact 379M‑parameter diffusion model that produces 1024‑pixel text‑to‑image results in about 1.4 seconds on a mobile device, achieving competitive FID scores and outperforming much larger models through a series of architecture refinements, advanced training tricks, and multi‑level knowledge distillation.
This article compiles recent AI research breakthroughs presented at AAAI 2025 and NeurIPS 2024, summarizing eight papers on multi‑condition image generation, mixed auto‑regressive models, hallucination mitigation in vision‑language models, quantized diffusion denoising, facial part swapping, language‑guided concept vectors, attribution consistency, and video virtual try‑on, with links to each work.
AniSora combines a 10‑million‑pair anime text‑video dataset, a controllable diffusion‑transformer with temporal‑mask conditioning for text‑to‑video, interpolation and region‑guided animation, and a 948‑video benchmark, delivering industry‑leading character and motion consistency and already powering low‑cost dynamic‑comic production for multiple IPs.
This work presents a subject‑driven diffusion pipeline that injects multi‑scale reference features (ReferenceNet‑style) into high‑fidelity backbones such as SD‑XL and Flux, enabling zero‑shot, fine‑grained product consistency across diverse scenes and outperforming current fine‑tuned and zero‑shot methods while noting limits in category coverage and human interactions.
EchoMimicV2, an open‑source project from Ant Group's Alipay AI team, introduces an end‑to‑end audio‑driven framework that generates high‑quality semi‑body portrait videos by jointly coordinating audio, pose, and image inputs, while addressing challenges of condition complexity, model stability, and computational cost.
FlowDCN introduces Groupwise‑MSDCN, a sparse deformable convolution that replaces attention, enabling efficient arbitrary‑resolution image generation with linear complexity, fewer parameters and FLOPs, and achieving state‑of‑the‑art FID scores on ImageNet while requiring far fewer training steps.
EchoMimicV2 is an open‑source digital‑human framework that generates high‑quality half‑body animation videos from a single reference image, an audio clip, and a hand‑gesture sequence, addressing challenges of facial portrait limits, complex condition injection, and inference latency in audio‑driven animation.
The paper proposes a multimodal Reliable Feedback Network (RFNet) and a consistency‑regularized fine‑tuning method (RFFT) that dramatically increase the proportion of usable advertising images generated by diffusion models while preserving visual appeal, and introduces the large‑scale RF1M dataset for training and evaluation.
The paper introduces a Multimodal Reliable Feedback Network (RFNet) and a consistency‑condition regularization technique that together boost the usable rate of automatically generated advertisement images while preserving visual quality, supported by a new million‑image annotated dataset and extensive ECCV‑2024 experiments.
This paper presents a multimodal Reliable Feedback Network (RFNet) and a consistency regularization method that use human feedback to automatically evaluate and fine‑tune diffusion models, dramatically increasing the usable rate of e‑commerce advertising images while preserving visual quality.
This paper presents a multimodal Trustworthy Feedback Network (RFNet) and a consistency regularization method that use human feedback to dramatically improve the usability and visual quality of automatically generated e‑commerce advertising images while reducing manual inspection costs.
MIT’s 6.S978 Deep Generative Models seminar, taught by Associate Professor He Kaiming, offers graduate students a 15‑week deep dive into VAEs, autoregressive models, GANs, diffusion techniques, and cross‑disciplinary applications, with detailed weekly topics, required assignments, and publicly available lecture PDFs.
This survey acts as a comprehensive portal that organizes AIGC research across seven domains—text, image, and audio generation, cross‑modal association, text‑guided image and audio synthesis, and supporting resources—detailing seminal models such as GPT, Diffusion, CLIP, DALL·E, Stable Diffusion, MusicLM, and key papers that shaped each field.
The VICTORIA algorithm, presented by Alibaba Cloud AI Platform PAI and South China University of Technology at ACM MM 2024, leverages linguistic dependency parsing to guide cross‑attention in Stable Diffusion, enabling accurate, training‑free multi‑object image editing while preserving spatial structure and achieving state‑of‑the‑art results on benchmark datasets.
The VICTORIA algorithm, presented by Alibaba Cloud's PAI team at ACM MM2024, leverages linguistic dependency parsing and cross‑attention control to overcome multi‑object editing challenges in training‑free text‑guided image editing, delivering precise, structure‑preserving results across diverse scenes.
Target‑Driven Distillation (TDD) is a multi‑goal distillation method that flexibly selects short‑range target steps and decouples guidance during training, enabling 4‑to‑8‑step diffusion generation that preserves high‑resolution detail, works with LoRA, ControlNet, InstantID, and outperforms existing consistency distillation techniques in speed and quality.
SPLAM (Sub‑Path Linear Approximation Model) accelerates diffusion‑model image synthesis by linearly approximating short sub‑paths of the probability‑flow ODE, allowing high‑quality generation in as few as four steps, outperforming prior fast‑sampling methods on COCO benchmarks and being deployed in Alibaba Mama’s recommendation system.
Recent ACL 2024 papers from Alibaba Cloud’s PAI platform showcase open‑source Chinese diffusion models, an interactive multi‑turn prompt generator, a long‑tail knowledge‑aware retrieval‑augmented LLM approach, and a dynamic fusion network for sequential model editing, all integrated into cloud services.
The article presents a comprehensive overview of Kuaishou’s Kolors (formerly 可图) multimodal generative model, detailing its data collection strategy, diffusion‑based architecture, evaluation metrics, derived capabilities such as prompt refinement and interactive generation, and a range of practical applications from AI‑powered live‑stream gifts to virtual try‑on, while also offering strategic advice for the domestic visual‑generation community.
MimicMotion is a controllable video generation framework that produces smooth, high-quality human motion videos by leveraging skeletal action guidance, addressing challenges in video generation such as limited length, weak controllability, and lack of dynamic detail.
EchoMimic, an open‑source project from Ant Group, presents a flexible, audio‑ and pose‑driven digital human generation pipeline that combines 2D, 3D and AIGC techniques, reduces production costs, achieves real‑time inference, and includes a detailed architecture, related work analysis, and future research directions.
This article explains the core principles of Stable Diffusion, its training pipeline and limitations, then details how ControlNet adds controllable signals to diffusion models, outlines its architecture, ecosystem of model variants, and showcases diverse real‑world applications.
Fast‑forwarding image‑to‑video generation, the article introduces I2V‑Adapter, a lightweight plug‑in for Stable Diffusion‑based video diffusion models that converts a single static image into a coherent video without altering the original T2V architecture, and details its design, frame‑similarity prior, experimental results, and real‑world applications.
The I2V-Adapter paper introduces a plug‑and‑play lightweight module that enables static images to be converted into dynamic videos using Stable Diffusion‑based text‑to‑video diffusion models without altering the original architecture or pretrained parameters, achieving competitive quality with far less training cost.
This article explains the nature of moiré patterns in digital imaging, reviews manual mitigation techniques, introduces direct and indirect AI‑based recognition methods—including traditional feature extraction and deep‑learning models such as CNNs and diffusion frameworks—and details practical applications and evaluation metrics used by Huolala.
This article reviews the rapid progress of text‑to‑video generation, explains diffusion‑based video synthesis, outlines key technical challenges such as motion modeling, semantic alignment and quality, and presents Tencent’s solutions and real‑world applications, while also discussing future directions and the impact of OpenAI’s Sora model.
Hyper‑SD introduces a trajectory‑segmented consistency distillation framework that combines trajectory‑preserving and trajectory‑reconstruction strategies, integrates human‑feedback learning and score distillation, and achieves state‑of‑the‑art low‑step image generation performance on both SD1.5 and SDXL models.
This article presents JD Advertising's 2023 AI-driven framework that uses a relation‑aware diffusion model with visual‑text and geometric modules, combined with category‑common and personalized generators and a planning‑and‑rendering network, to automate high‑quality, scalable e‑commerce poster creation and background synthesis.
The article reviews the latest breakthroughs in generative AI, including Microsoft’s VASA‑1 video synthesis model, Meta’s open‑source Llama‑3 large language model, Stability AI’s Stable Diffusion 3 API, Adobe’s integration of third‑party AI video tools into Premiere Pro, and a free image‑style‑recreation platform from Freepik, highlighting their technical details and potential applications.
Since 2023, AI‑generated video research has expanded across six algorithmic categories—text‑to‑video, image‑to‑video, editing, style transfer, human motion, and long‑video generation—highlighting works such as CogVideo, Imagen Video, MagicVideo, ControlVideo, DCTNet, NUWA‑XL and OpenAI’s Sora, while analysis shows short‑clip diffusion models excel, editing remains costly, style transfer is efficient, and truly long, temporally consistent videos remain an open challenge.
This comprehensive review examines the rapid advances in generative AI for multimodal image synthesis and editing, covering visual, textual, and audio guidance, model families such as GANs, diffusion, autoregressive, and NeRF, as well as datasets, challenges, and future research directions.
The article provides a comprehensive technical overview of OpenAI’s Sora text‑to‑video model, explaining its background, underlying diffusion‑Transformer architecture, key breakthroughs, potential industry impacts, success factors, limitations, and future prospects for AI‑generated video content.
This article reviews OpenAI's Sora video generation model, summarizing its technical report, key innovations such as patch-based visual tokens, compression networks, scaling transformers, language understanding, and discussing its capabilities, highlights, and current limitations in AI video synthesis.
OpenAI’s Sora model demonstrates large‑scale text‑conditional video generation using a diffusion transformer that operates on spatiotemporal patches, supporting variable durations, resolutions, and aspect ratios while showcasing emergent simulation abilities, flexible sampling, and multimodal editing capabilities, though it still has notable limitations.
OpenAI’s newly unveiled Sora model transforms short text prompts into up‑to‑one‑minute high‑definition videos, showcasing advanced diffusion‑Transformer architecture, improved occlusion handling, and detailed visual fidelity, while the article examines its technical breakthroughs, compares it to earlier models, and discusses emerging safety and misuse concerns.
InstantID is a plug‑in adapter that adds identity‑preserving capabilities to text‑to‑image diffusion models, allowing users to upload a single photo and, within 30 seconds, produce a Spring Festival‑styled portrait with accurate facial features, customizable prompts, and strong text control.
The Hierarchical Masked 3D Diffusion Model (M3DDM) introduces a masking‑based training strategy and cross‑attention with global video clips to achieve temporally consistent video outpainting, while a hybrid coarse‑to‑fine inference pipeline mitigates error accumulation, delivering state‑of‑the‑art results and deployment in Alibaba’s creative center.
The second Baidu Search Innovation Contest attracted over 2,800 participants from 45 regions, featured five AI‑focused tracks, and highlighted champion teams that employed techniques such as Lora‑fine‑tuned LLMs, vector‑intersection Top‑K search, GPU‑optimized algorithms, and diffusion‑based image generation to push the boundaries of search technology.
The AIGC Poster Generation Project employs Stable Diffusion enhanced with VAE, ControlNet, LoRA and other extensions to create product posters in four visual styles, exploring outpainting, inpainting, reference‑based diffusion and DreamBooth prototypes, and optimizes detail preservation, super‑resolution text, and masking to achieve over 90% detail fidelity, 95% success rate, and 3–5 second inference per image.
This article reviews recent progress in advertising recommendation technologies, covering deep learning‑based ranking, sequence modeling, self‑supervised learning, online and reinforcement learning, multimodal recommendation, and fairness, and details four key breakthroughs—data‑driven incremental learning, dynamic group parameter modeling, bilateral interactive graph convolution, and a relation‑aware diffusion model for poster layout generation, along with experimental results and future challenges.
This article explains how contextual learning can enhance text‑to‑image models by incorporating example image‑text pairs, redesigning the UNet architecture, building large in‑context training datasets, and training the SuTI model to achieve fast, controllable, and high‑quality personalized image generation.
Google, in collaboration with UC Berkeley, has unveiled a novel generative AI approach called the Idempotent Generative Network (IGN) that can produce images from any input in a single step, offering an alternative to GANs, diffusion models, and consistency models.
This article explores powerful AIGC generation models and large language models like ChatGPT, detailing how diffusion models can be applied to robotic planning, introducing AdaptDiffuser, self‑evolving data generation, and embodied AI challenges, while summarizing recent research and practical implementations.
Professor Hu Qiang presented recent progress in Neural Radiance Fields—covering implicit/explicit representations, hybrid models, and solutions for dynamic scenes, cloud‑based and edge‑cloud rendering—while also reviewing generative AI advances such as diffusion‑based text‑to‑image/video/3D, LoRA fine‑tuning, and large‑scale story‑book datasets, highlighting applications in virtual‑real content, smart‑city modeling, and 6‑DoF e‑commerce displays.
This article presents the OLSS (Optimal Linear Subspace Search) algorithm, a novel diffusion‑model sampling accelerator that models acceleration as a linear subspace expansion, unifies existing methods, introduces trainable scheduler coefficients solved via least‑squares, and demonstrates significant speed and quality gains on Stable Diffusion benchmarks.
The article announces that Alibaba Cloud’s AI platform PAI and ECNU researchers’ paper on the Optimal Linear Subspace Search (OLSS) algorithm was selected for CIKM 2023, explains how OLSS accelerates diffusion‑model sampling by operating in higher‑dimensional linear subspaces, and provides details of the paper and its visual results.
The article presents DiffUTE, an end‑to‑end self‑supervised multilingual text‑editing diffusion model that leverages fine‑grained position and glyph guidance together with large language model control to achieve high‑quality, high‑fidelity text modifications in images, and demonstrates its effectiveness through extensive experiments and real‑world deployments at Ant Group.
This article explores the rise of AI‑driven design, examines real‑world projects such as AI‑Soup, virtual jewelry, and e‑commerce checkpoints, explains diffusion model fundamentals, compares major tools, and discusses the opportunities and risks for creators and businesses adopting generative AI.
This article reviews four ACM MM 2023 papers that introduce AI‑driven systems for automatic advertising poster generation, multimodal text‑image creation, few‑shot style‑guided visual captioning, and hierarchical 3D diffusion models for video outpainting, detailing their methods, datasets, and experimental results.
Three papers from Alibaba Cloud's PAI platform were selected for ACL 2023, showcasing FashionKLIP for e‑commerce image‑text retrieval, ConaCLIP's lightweight dual‑encoder distillation, and a fast domain‑specific diffusion model, all of which will be open‑sourced for the AI community.
This article provides a comprehensive overview of Stable Diffusion, covering its latent diffusion architecture, training data and procedures, model components such as autoencoder, CLIP text encoder and UNet, as well as practical usage examples including text‑to‑image generation, image‑to‑image, inpainting, and advanced extensions like ControlNet and SD‑2.x.
This guide explains how AI creates photorealistic images, tracing the shift from VAEs and GANs to diffusion models, detailing latent diffusion, ControlNet conditioning, CLIP text‑image alignment, and lightweight fine‑tuning methods like DreamBooth and LoRA, plus practical tips for higher‑resolution results.
This article surveys the key challenges of controllable text‑to‑image generation and explains the architectures, components, and training details of major diffusion‑based models such as DALL‑E‑2, Google Imagen, Stability AI's Latent Stable Diffusion, and the ControlNet extension.
This article reviews the key components and design choices of recent text‑controlled image generation systems—including DALL·E‑2, Google Imagen, Stability AI's Latent Stable Diffusion, and the ControlNet extension—highlighting how diffusion models, text encoders, prior modules, super‑resolution, and conditioning mechanisms enable high‑quality, controllable visual synthesis.
This article introduces Alibaba Cloud’s open‑source PAI‑Diffusion series, detailing its Latent Diffusion Model foundation, Chinese CLIP alignment, super‑resolution components, and showcases diverse artistic and real‑world text‑to‑image generation scenarios, while providing guidance on accessing the models via Alibaba Cloud AI Center, PAI‑DSW, and HuggingFace Space.
The article compares Google’s Imagen and OpenAI’s DALL·E by feeding them the whimsical "Tiger Wearing VR" prompt, showcasing each model’s visual style, underlying architecture—including CLIP, diffusion, and T5‑XXL—and community reactions to the resulting AI‑generated artwork.
This article explains the fundamentals of denoising diffusion probabilistic models, detailing the forward Gaussian noise injection, the reverse reconstruction via learned conditional densities, model architecture, loss functions, and experimental results on synthetic datasets, all supported by key research citations.
