通过注意力引导降噪与动态令牌选择,提升文本图像一致性并降低计算开销。
OptiPrune: Boosting Prompt-Image Consistency with Attention-Guided Noise and Dynamic Token Selection
- 用注意力分数指导初始噪声优化,聚焦语义关键区域。
- 基于相似性动态筛选令牌,减少计算量同时保持生成质量。
- 适合部署在资源受限设备上的高效文生图模型设计。
文本到图像扩散模型常因难以实现生成图像与文本提示间的精准语义对齐,且在资源受限硬件上部署效率低下。现有方法或因噪声优化带来显著计算开销,或因过度剪枝牺牲语义保真度。本文提出OptiPrune统一框架,结合分布感知的初始噪声优化与基于相似性的令牌剪枝,同步解决上述问题。具体而言:(1) 设计基于注意力得分的分布感知噪声优化模块,引导初始潜在噪声向语义有意义区域迁移,缓解主体忽略与特征纠缠;(2) 提出硬件友好的令牌剪枝策略,通过逐块相似性选择代表性基础令牌,注入随机性增强泛化能力,并在注意力前以最大相似性复制恢复被剪枝令牌。该方法在噪声优化中保留高斯先验,实现无损推理效率。在Animal-Animal等基准数据集上实验表明,OptiPrune在显著降低计算成本的同时达到最先进的提示-图像一致性性能。
原文摘要 · Abstract (English)
Text-to-image diffusion models often struggle to achieve accurate semantic alignment between generated images and text prompts while maintaining efficiency for deployment on resource-constrained hardware. Existing approaches either incur substantial computational overhead through noise optimization or compromise semantic fidelity by aggressively pruning tokens. In this work, we propose OptiPrune, a unified framework that combines distribution-aware initial noise optimization with similarity-based token pruning to address both challenges simultaneously. Specifically, (1) we introduce a distribution-aware noise optimization module guided by attention scores to steer the initial latent noise toward semantically meaningful regions, mitigating issues such as subject neglect and feature entanglement; (2) we design a hardware-efficient token pruning strategy that selects representative base tokens via patch-wise similarity, injects randomness to enhance generalization, and recovers pruned tokens using maximum similarity copying before attention operations. Our method preserves the Gaussian prior during noise optimization and enables efficient inference without sacrificing alignment quality. Experiments on benchmark datasets, including Animal-Animal, demonstrate that OptiPrune achieves state-of-the-art prompt-image consistency with significantly reduced computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。