将注意力机制视为向量符号计算，解释模型推理与失效原因
Attention as Binding: A Vector-Symbolic Perspective on Transformer Reasoning
- 用向量符号架构解析注意力机制，查询/键为角色，值为填充物
- 揭示变量混淆、逻辑不一致等失败模式的内在根源
- 提出绑定解绑头和高维记忆层，提升逻辑可靠性
基于Transformer的语言模型展现出令人印象深刻的类推理行为，但在需要稳定符号操作的任务上仍显脆弱。本文通过将自注意力和残差流视为近似向量符号架构（VSA）的实现，建立统一视角：查询与键定义角色空间，值编码填充物，注意力权重执行软解绑，残差连接实现多重绑定结构的叠加。该代数视角将模型内部机制与思维链轨迹、程序化推理及记忆增强工具使用联系起来，并解释了变量混淆、逻辑相关提示间的不一致性等典型失败模式。在此基础上，提出受VSA启发的架构偏置，包括显式绑定/解绑头和超维记忆层，以及促进角色-填充分离与鲁棒叠加的训练目标。最后，提出衡量“VSA相似性”与逻辑组合性的度量指标，并指出理论与架构上的开放问题。总体而言，将注意力视为软向量符号计算，为构建更可解释、逻辑可靠的推理系统提供了原则性路径。
原文摘要 · Abstract (English)
Transformer-based language models display impressive reasoning-like behavior, yet remain brittle on tasks that require stable symbolic manipulation. This paper develops a unified perspective on these phenomena by interpreting self-attention and residual streams as implementing an approximate Vector Symbolic Architecture (VSA). In this view, queries and keys define role spaces, values encode fillers, attention weights perform soft unbinding, and residual connections realize superposition of many bound structures. We use this algebraic lens to relate transformer internals to chain-of-thought traces, program-based reasoning, and memory-augmented tool use, and to explain characteristic failure modes such as variable confusion and inconsistency across logically related prompts. Building on this perspective, we propose VSA-inspired architectural biases, including explicit binding/unbinding heads and hyperdimensional memory layers, and training objectives that promote role-filler separation and robust superposition. Finally, we outline metrics for measuring "VSA-likeness" and logical compositionality, and pose theoretical and architectural open problems. Overall, the paper argues that viewing attention as soft vector-symbolic computation offers a principled route toward more interpretable and logically reliable reasoning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。