通过感知美学注意力,让扩散模型生成更快更美。
AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generation
- 根据提示语和注意力信号生成美学聚焦掩码,只在关键区域计算
- 在Lumina-Next上实现2.11倍加速,ImageReward提升11.9%
- 无需训练,适合追求高效高质图像生成的开发者
扩散Transformer(DiTs)因其在高分辨率下良好的可扩展性和对齐能力,成为高质量文生图生成的主流骨干网络。然而,密集空间标记上的二次自注意力导致推理延迟高,限制了部署。我们观察到去噪过程在美学描述符上存在空间非均匀性:与美学标记相关的区域接收集中交叉注意力且随时间变化大,而低关联区域演化平滑且计算冗余。基于此,提出AccelAes——一种无需训练的加速框架,通过美学感知的时空剪枝提升推理速度并改善感知美学。AccelAes构建AesMask,一种从提示语义和交叉注意力信号生成的一次性美学聚焦掩码;当局部计算可行时,SkipSparse将计算和引导重新分配至掩码区域。进一步通过轻量级步级预测缓存减少时间冗余,定期替代完整Transformer评估。在代表性DiT模型上实验显示一致加速与美学质量提升。在Lumina-Next上,AccelAes实现2.11×加速,ImageReward相较密集基线提升+11.9%。代码已开源。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) are a dominant backbone for high-fidelity text-to-image generation due to strong scalability and alignment at high resolutions. However, quadratic self-attention over dense spatial tokens leads to high inference latency and limits deployment. We observe that denoising is spatially non-uniform with respect to aesthetic descriptors in the prompt. Regions associated with aesthetic tokens receive concentrated cross-attention and show larger temporal variation, while low-affinity regions evolve smoothly with redundant computation. Based on this insight, we propose AccelAes, a training-free framework that accelerates DiTs through aesthetics-aware spatio-temporal reduction while improving perceptual aesthetics. AccelAes builds AesMask, a one-shot aesthetic focus mask derived from prompt semantics and cross-attention signals. When localized computation is feasible, SkipSparse reallocates computation and guidance to masked regions. We further reduce temporal redundancy using a lightweight step-level prediction cache that periodically replaces full Transformer evaluations. Experiments on representative DiT families show consistent acceleration and improved aesthetics-oriented quality. On Lumina-Next, AccelAes achieves a 2.11$\times$ speedup and improves ImageReward by +11.9% over the dense baseline. Code is available at https://github.com/xuanhuayin/AccelAes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。