通过静态文本锚点实现参考图像高效重用，显著加速多参考图像编辑。
Anchoring Instruction Outside Mask: Exact Reference Caching for Efficient In-Context Diffusion Transformers
- 设计静态文本锚点连接指令与参考分支，支持精确键值复用。
- 五张参考图下提速3.92倍，十张时达5.47倍，生成质量接近全注意力模型。
- 首次在扩散模型中使用在线策略蒸馏修复架构改造带来的性能损失。
跨模态生成是内容创作与编辑的核心。上下文条件化使扩散变压器能将文本指令与视觉参考置于共享注意力序列中处理。然而，每张参考图引入数千个标记，计算量随参考数快速增长。现有方法采用结构化稀疏注意力以减少计算，但限制了参考与目标标记间的交互，且使参考的键（K）和值（V）独立于去噪目标，可一次性计算并复用。这导致视觉参考无法关注文本指令，严重降低多参考编辑中的指令遵循与参考保真度。为解决这一矛盾，我们联合重构标记序列与注意力掩码。超越掩码的设计使用静态文本锚点连接指令与参考分支，保留精确的K、V复用且不增加参数。然而，这种直接架构转换会降低生成质量。我们通过教师强制速度蒸馏，再辅以短时在线策略阶段，由教师监督学生状态，恢复性能。据我们所知，这是首个在扩散模型中使用在线策略蒸馏进行架构修复的工作。在三个图像编辑基准上，该方法达到全注意力生成质量；含五张参考图时，40步去噪过程加速3.92倍，静态文本锚点引入的运行时开销可忽略；在扩展研究中，十张参考图时加速达5.47倍。
原文摘要 · Abstract (English)
Omnimodal generation is central to a wide range of content creation and editing applications. In-context conditioning is essential to this paradigm. It allows diffusion transformers to process text instructions and visual references in a shared attention sequence. However, each reference image introduces thousands of tokens. Computation therefore grows rapidly with the number of references. Existing methods reduce computation through structured sparse attention, which limits interactions between reference and target tokens. This structure also makes the reference K and V independent of the denoising target, allowing them to be computed once and reused across steps. However, it blocks visual references from attending to the text instruction. This substantially degrades instruction following and reference fidelity in multi-reference editing. To resolve this conflict, we jointly redesign the token sequence and attention mask. Our beyond-mask design uses static text anchors to connect the instruction to the reference branch. It preserves exact K and V reuse without adding parameters. However, this direct architectural conversion degrades generation quality. We recover the lost performance through teacher-forced velocity distillation, followed by a short on-policy stage in which the teacher supervises student-visited states. To our knowledge, this is the first use of on-policy distillation for architectural recovery in diffusion models. Across three image-editing benchmarks, our method matches full-attention generation quality. With five reference images, it accelerates the complete 40-step denoising process by 3.92x, while static text anchors introduce negligible runtime overhead; the speedup reaches 5.47x at ten references in our scaling study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。