通过打破注意力机制对称性，提升模型性能与可解释性
Symmetry Breaking in Transformers for Efficient and Interpretable Training
- 引入批量采样的固定偏置，打破注意力的旋转对称性
- 使简单优化器性能接近复杂自适应方法，验证损失降低30%以上
- 可识别并放大特定语义词元，适合需要透明决策的场景
标准注意力机制存在冗余的旋转自由度，这些自由度在计算中传递但不影响激活或输出。本文提出一种简单的对称性破缺策略：通过批量采样、不可学习的查询和值偏置引入偏好方向。该修改带来两个理论支持且实验验证的结果：首先，显著提升简单、低内存优化器的性能，缩小甚至消除其与复杂高内存自适应方法（如AdamW）的差距；我们在124M参数Transformer上用四种算法（AdamW、SOAP、SGDM、ECD）预训练，并评估验证损失与下游逻辑推理能力。其次，使原本冗余的旋转自由度变得可解释，能选择性增强单个注意力头中的语义相关词元类别。结果表明，极小的、有原则的架构改动可同时提升性能与可解释性。
原文摘要 · Abstract (English)
The attention mechanism in its standard implementation contains extraneous rotational degrees of freedom that are carried through computation but do not affect model activations or outputs. We introduce a simple symmetry-breaking protocol that inserts a preferred direction into this rotational space through batchwise-sampled, unlearned query and value biases. This modification has two theoretically motivated and empirically validated consequences. First, it can substantially improve the performance of simple, memory-efficient optimizers, narrowing -- and in some cases closing -- the gap to successful but more complex memory-intensive adaptive methods. We demonstrate this by pretraining 124M parameter transformer models with four optimization algorithms (AdamW, SOAP, SGDM, and Energy Conserving Descent(ECD)) and evaluating both validation loss and downstream logical reasoning. Second, it enables an interpretable use of otherwise redundant rotational degrees of freedom, selectively amplifying semantically meaningful token classes within individual attention heads. Overall, our results show that minimal, principled architectural changes can simultaneously improve performance and interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。