arXiv:2602.06850cs.CVcs.AI2026-02

提出PKA框架,让多条件图像生成更快更省显存。

Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping

  • 用位置对齐和关键词裁剪,减少冗余注意力计算。
  • 推理速度提升10倍,显存占用减少5.1倍。
  • 适合需要精细控制的图像生成场景,如布局与人物特征调节。

现代文本到图像模型虽能基于提示生成图像,但在空间布局或主体外观等细粒度控制方面仍显不足。多条件控制可解决此问题,但其在扩散变换器(DiTs)中的集成受限于传统的“拼接-注意力”策略,该策略随条件数量增加导致计算和内存开销呈二次增长。我们分析发现,大量跨模态交互在空间或语义上存在冗余。为此,提出位置对齐与关键词裁剪注意力(PKA)框架,以消除这些冗余。具体而言,位置对齐注意力(PAA)通过强制局部块对齐实现空间控制的线性化;关键词裁剪注意力(KSA)则通过语义感知掩码剔除无关的主体驱动交互。为促进高效学习,进一步引入条件敏感性感知采样(CSAS)策略,重新加权训练目标至关键去噪阶段,显著加速收敛并提升条件保真度。实验表明,PKA实现10.0×推理加速和5.1×显存节省,为高保真多条件生成提供可扩展、资源友好的解决方案。

原文摘要 · Abstract (English)

While modern text-to-image models excel at prompt-based generation, they often lack the fine-grained control necessary for specific user requirements like spatial layouts or subject appearances. Multi-condition control addresses this, yet its integration into Diffusion Transformers (DiTs) is bottlenecked by the conventional ``concatenate-and-attend'' strategy, which suffers from quadratic computational and memory overhead as the number of conditions scales. Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant. To this end, we propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies. Specifically, Position-Aligned Attention (PAA) linearizes spatial control by enforcing localized patch alignment, while Keyword-Scoped Attention (KSA) prunes irrelevant subject-driven interactions via semantic-aware masking. To facilitate efficient learning, we further introduce a Conditional Sensitivity-Aware Sampling (CSAS) strategy that reweights the training objective towards critical denoising phases, drastically accelerating convergence and enhancing conditional fidelity. Empirically, PKA delivers a 10.0$\times$ inference speedup and a 5.1$\times$ VRAM saving, providing a scalable and resource-friendly solution for high-fidelity multi-conditioned generation.

图像生成扩散模型注意力机制高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。