通过稀疏注意力提升扩散模型推理性能，无需额外训练
PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference Time by Leveraging Sparsity
- 利用软最大化与稀疏注意力的关联外推，改进交叉注意力
- 在不增加计算量情况下显著提升文本对齐与人类偏好评分
- 兼容蒸馏引导模型，适合部署于现有扩散系统
扩散模型在条件生成任务中表现优异，常依赖分类器无关引导（CFG）等技术。然而现有方法多需额外训练或神经网络函数评估（NFE），与蒸馏引导模型不兼容，且依赖启发式策略确定目标层。本文提出新方法PLADIS，通过推理阶段利用交叉注意力层中的稀疏注意力特性，无需额外训练或NFE，即可增强预训练模型（如U-Net/Transformer）性能。通过外推查询-键相关性，结合稀疏注意力的抗噪优势，充分释放文生图扩散模型的潜在能力，在原本表现较弱的领域实现显著提升。该方法可无缝集成至各类引导技术，包括蒸馏模型。大量实验表明，其在文本对齐和人类偏好上均有明显改善，提供高效且通用的解决方案。
原文摘要 · Abstract (English)
Diffusion models have shown impressive results in generating high-quality conditional samples using guidance techniques such as Classifier-Free Guidance (CFG). However, existing methods often require additional training or neural function evaluations (NFEs), making them incompatible with guidance-distilled models. Also, they rely on heuristic approaches that need identifying target layers. In this work, we propose a novel and efficient method, termed PLADIS, which boosts pre-trained models (U-Net/Transformer) by leveraging sparse attention. Specifically, we extrapolate query-key correlations using softmax and its sparse counterpart in the cross-attention layer during inference, without requiring extra training or NFEs. By leveraging the noise robustness of sparse attention, our PLADIS unleashes the latent potential of text-to-image diffusion models, enabling them to excel in areas where they once struggled with newfound effectiveness. It integrates seamlessly with guidance techniques, including guidance-distilled models. Extensive experiments show notable improvements in text alignment and human preference, offering a highly efficient and universally applicable solution. See Our project page : https://cubeyoung.github.io/pladis-proejct/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。