通过分解注意力矩阵，实现生成模型在保真度与多样性间的可控平衡。
Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective
- 将注意力矩阵分解为对称与反对称部分，分别对应能量景观与动态循环。
- 提出霍普菲尔德稳定性度量，其与生成质量的保真度-多样性权衡显著相关。
- 可调节动态循环强度，实现对生成结果多样性的灵活控制。
我们把Transformer中的预软最大注意力矩阵$\mathbf{QK^\top}$视为编码输入特征间成对关联的关联记忆矩阵。通过将其分解为对称与反对称部分，解释对称分量主导能量景观结构，反对称分量驱动该景观上的循环动力学。基于对称分量诱导的能量形式，我们推导出类似霍普菲尔德的稳定性度量，用于量化检索特征的稳定性。观察到该稳定性度量与生成过程中的保真度-多样性权衡存在显著相关性。最后，我们提出一种可调控的机制，通过修改底层动力学的循环性来调节这一权衡。代码已开源于GitHub（https://github.com/hyeon-cho/Attention-Symmetric-Decomposition）。
原文摘要 · Abstract (English)
We characterize the pre-softmax attention matrix $\mathbf{QK^\top}$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric parts, we interpret the symmetric component as governing the structure of the energy landscape, and the skew-symmetric component as driving circulation on that landscape. Leveraging the energy formulation induced by the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features. We observe meaningful correlations between Hopfield-style stability measures and the fidelity-diversity trade-offs in generation. Finally, we propose a controllable knob to modulate this trade-off by modifying the circulation of the underlying dynamics. Code is available at our GitHub (https://github.com/hyeon-cho/Attention-Symmetric-Decomposition).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。