让编码器模型自适应调整注意力,提升推理能力而不增加参数
Change of Thought: Adaptive Test-Time Computation
- 通过迭代优化自身注意力权重,实现测试时自适应计算
- 在编码器任务上最高提升20%准确率,无需额外参数
- 适合追求高效高精度推理的场景,尤其适合资源受限部署
单次固定深度评估的Transformer在表达能力上被证明受限于TC0常数深度电路类。而自回归运行的Transformer则突破此限制——从下一个词预测到思维链推理均依赖反馈机制,将内部状态解码为标记再重新编码。这种“自言自语”虽类比人类思维,但生物大脑迭代时不以外化中间状态为语言。为在不使用标记级自回归的前提下增强编码器Transformer的表达力,我们提出SELF-Transformer:一种迭代更新自身注意力权重至不动点的编码器层。不再一次生成输入序列的对齐矩阵,而是内部持续迭代优化该矩阵,使测试时计算量随输入难度动态增长。这种自适应性在编码器基准任务上带来高达20%的准确率提升,且不增加参数量,表明测试时输入自适应对齐仅需少量额外计算开销即可带来显著收益。因此,Self-Transformers在保持纯编码器架构简洁性的前提下,恢复了大量迭代推理的表达能力。
原文摘要 · Abstract (English)
Transformers evaluated in a single, fixed-depth pass are provably limited in expressive power to the constant-depth circuit class TC0. Running a Transformer autoregressively removes that ceiling -- first in next-token prediction and, more recently, in chain-of-thought reasoning. Both regimes rely on feedback loops that decode internal states into tokens only to re-encode them in subsequent steps. While this "thinking aloud" mirrors human reasoning, biological brains iterate without externalising intermediate states as language. To boost the expressive power of encoder Transformers without resorting to token-level autoregression, we introduce the SELF-Transformer: an encoder layer that iteratively refines its own attention weights to a fixed point. Instead of producing -- in one pass -- the alignment matrix that remixes the input sequence, the SELF-Transformer iteratively updates that matrix internally, scaling test-time computation with input difficulty. This adaptivity yields up to 20\% accuracy gains on encoder-style benchmarks without increasing parameter count, demonstrating that input-adaptive alignment at test time offers substantial benefits for only a modest extra compute budget. Self-Transformers thus recover much of the expressive power of iterative reasoning while preserving the simplicity of pure encoder architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。