用多模态大模型做图像去噪，让生成更准更快
RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space
- 用多模态大模型直接编码含噪视觉表示，替代传统去噪器
- 在相同计算量下，生成质量优于新训练的去噪模型
- 适合想提升文本到图像生成效率的研究者和开发者
大型语言模型（LLM）广泛用于文本到图像（T2I）系统，但通常仅负责文本编码，而去噪由新训练的生成主干处理。表示自编码器（RAE）将生成目标转向语义结构化的视觉表示，形成与预训练LLM先验更兼容的潜在空间。受多模态LLM（MLLM）启发，其中仅需一个MLP投影器即可对齐干净视觉表示与预训练LLM，我们重新利用MLLM本身作为含噪表示编码器，将该机制从干净输入扩展至含噪输入。本文提出RepFusion，使用所得MLLM输出作为扩散变换器的条件信号。在相似推理预算下的控制对比中，RepFusion优于将相当算力投入新初始化去噪器的基线模型。结果表明，MLLM为视觉表示去噪提供了强先验，且通过在测试时反复对含噪表示进行MLLM条件化，可高效利用计算资源。
原文摘要 · Abstract (English)
Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. These results demonstrate that MLLMs provide strong priors for denoising visual representations and that, by conditioning on evolving noisy representations, test-time compute can be productively spent on repeated MLLM conditioning in modern T2I systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。