用扩散模型修正自回归生成的错误,实现高质量多模态内容生成。
ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correction
- 自回归模型生成全局结构,扩散模型局部修正误差
- 零样本适配,不需微调即可提升生成质量
- 适合需要长序列、高保真多模态生成的场景
自回归模型(ARMs)擅长建模全局上下文和长序列生成,扩散模型(DMs)则在生成高质量局部内容方面表现优异,尤其适用于图像和短视频等连续数据。然而,自回归模型在长序列中会累积指数级错误,导致物理上不合理的结果;而扩散模型受限于局部上下文生成能力。本文提出一种无需微调的零样本方法——基于扩散校正的自回归一致多模态生成(ACDC),在推理阶段结合两者优势:利用自回归模型生成全局结构,通过记忆增强的扩散模型对生成的多模态标记进行局部修正,消除伪影。我们设计了一个基于大语言模型的动态记忆模块,根据全局语境调整扩散模型的条件输入,保留关键上下文信息。在多模态任务(包括连贯多帧故事生成和自回归视频生成)上的实验表明,ACDC有效缓解了错误累积,显著提升生成质量,性能优于基线,且对具体模型架构无依赖。项目页面:https://acdc2025.github.io/
原文摘要 · Abstract (English)
Autoregressive models (ARMs) and diffusion models (DMs) represent two leading paradigms in generative modeling, each excelling in distinct areas: ARMs in global context modeling and long-sequence generation, and DMs in generating high-quality local contexts, especially for continuous data such as images and short videos. However, ARMs often suffer from exponential error accumulation over long sequences, leading to physically implausible results, while DMs are limited by their local context generation capabilities. In this work, we introduce Autoregressive Coherent multimodal generation with Diffusion Correction (ACDC), a zero-shot approach that combines the strengths of both ARMs and DMs at the inference stage without the need for additional fine-tuning. ACDC leverages ARMs for global context generation and memory-conditioned DMs for local correction, ensuring high-quality outputs by correcting artifacts in generated multimodal tokens. In particular, we propose a memory module based on large language models (LLMs) that dynamically adjusts the conditioning texts for the DMs, preserving crucial global context information. Our experiments on multimodal tasks, including coherent multi-frame story generation and autoregressive video generation, demonstrate that ACDC effectively mitigates the accumulation of errors and significantly enhances the quality of generated outputs, achieving superior performance while remaining agnostic to specific ARM and DM architectures. Project page: https://acdc2025.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。