arXiv:2507.12771cs.CVcs.AI2025-07被引 1

提出新合并策略,让文生图模型更快更准

Local Representative Token Guided Merging for Text-to-Image Generation

  • 按局部窗口动态合并令牌,保留关键视觉特征
  • FID提升6.2%,CLIP分数更高,推理时间不变
  • 适配各类注意力机制,适合追求效率的生成应用

Stable Diffusion 是优秀的文生图模型,但其注意力计算的二次复杂度导致生成速度慢。现有令牌合并方法常忽略注意力生成模型特性,效果受限。本文提出局部代表性令牌引导合并(ReToM),适用于任意图像生成中的注意力机制。ReToM通过定义注意力输入内的局部窗口并自适应调整窗口大小,结合特定时刻的相似性计算,选出每个窗口中平均相似度最高的代表令牌,从而在减少令牌数的同时保留最显著的局部特征,降低计算开销。实验表明,ReToM相比基线在FID上提升6.2%,CLIP分数更高,且推理时间相当。实证显示,ReToM能有效平衡视觉质量与计算效率。

原文摘要 · Abstract (English)

Stable diffusion is an outstanding image generation model for text-to-image, but its time-consuming generation process remains a challenge due to the quadratic complexity of attention operations. Recent token merging methods improve efficiency by reducing the number of tokens during attention operations, but often overlook the characteristics of attention-based image generation models, limiting their effectiveness. In this paper, we propose local representative token guided merging (ReToM), a novel token merging strategy applicable to any attention mechanism in image generation. To merge tokens based on various contextual information, ReToM defines local boundaries as windows within attention inputs and adjusts window sizes. Furthermore, we introduce a representative token, which represents the most representative token per window by computing similarity at a specific timestep and selecting the token with the highest average similarity. This approach preserves the most salient local features while minimizing computational overhead. Experimental results show that ReToM achieves a 6.2% improvement in FID and higher CLIP scores compared to the baseline, while maintaining comparable inference time. We empirically demonstrate that ReToM is effective in balancing visual quality and computational efficiency.

文生图令牌合并Stable Diffusion效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。