提出早期内容对齐评估方法，降低扩散模型生成成本。
Early Estimation of Language to Latent Alignment in Diffusion Models
- 用噪声感知孪生网络在生成中途评估图文对齐度
- 半程成本实现效果优于全程冻结CLIP
- 适合需要高效采样的图像生成场景
条件扩散模型常出现语言与图像对齐偏差。由于中间噪声扰动的潜在表示模糊，当前评估提示遵循性需完成整个采样过程，导致测试时扩展策略（如Best-of-N）计算开销剧增。为此，本文提出NoisyCLIP，一种噪声感知的双塔模型，可在生成早期实现语言到潜在空间的对齐估计。通过在噪声扰动的潜在表示上训练视觉编码器，使模型能“穿透”中间步骤的模糊性。为支持训练，研究了噪声数据增强采样策略，并构建两个新基准数据集：Noisy-Conceptual-Captions和Noisy-GenAI-Bench。当作为BoN的早期停止准则时，NoisyCLIP仅需一半计算成本即可达到或超越全成本冻结CLIP的效果。该方法将对齐评估从高成本最终检查转变为持续监控工具，显著降低计算开销且不损失语义保真度。
原文摘要 · Abstract (English)
Conditional diffusion models frequently suffer from language-image misalignments. Due to the ambiguity of intermediate noise corrupted latents, assessing prompt adherence currently requires completing the entire sampling trajectory. This late-stage evaluation incurs even higher computational costs during test-time scaling strategies, such as Best-of-N (BoN) sampling, as all misaligned trajectories must finish generation before being discarded. To tackle this, we propose NoisyCLIP, a noise-aware twin-tower model that enables early language-to-latent alignment estimation. By learning a vision encoder on noise-corrupted latents, we allow the model to "see" through the ambiguity of intermediate diffusion steps. To facilitate this training, we investigate noise-data augmentation sampling strategies and introduce two new benchmark datasets: Noisy-Conceptual-Captions and Noisy-GenAI-Bench. When applied as an early-stopping criterion for BoN, NoisyCLIP at half cost matches or beats frozen CLIP at full cost. Ultimately, this transforms alignment assessment from an expensive final check into a continuous monitoring tool, drastically reducing compute costs without sacrificing semantic fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。