揭示大模型内部如何表示运算符优先级，发现中间结果可被追踪。
Interpreting the Latent Structure of Operator Precedence in Language Models
- 通过残差流分析，发现模型在MLP层后保留中间计算结果。
- 运算符嵌入在注意力层后线性编码优先级顺序。
- 提出部分嵌入交换法，可主动改变运算优先级逻辑。
大型语言模型虽具备出色推理能力，但在算术任务上仍表现不佳。现有研究多关注输出或提示策略，却未探究模型内部如何进行算术计算。本文以开源指令微调的LLaMA 3.2-3B模型为对象，构建包含三个操作数和两个运算符、括号位置与顺序各异的算术表达式数据集，通过日志透镜、线性分类探针及UMAP几何可视化等可解释性技术，追踪模型残差流中是否出现中间结果。结果显示，中间计算确实在残差流中存在，尤其在MLP块之后；同时，每个运算符的嵌入在注意力层后线性编码了其优先级。本文提出局部嵌入交换技术，通过交换高影响力嵌入维度，实现对运算优先级的主动调控。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive reasoning capabilities but continue to struggle with arithmetic tasks. Prior works largely focus on outputs or prompting strategies, leaving the open question of the internal structure through which models do arithmetic computation. In this work, we investigate whether LLMs encode operator precedence in their internal representations via the open-source instruction-tuned LLaMA 3.2-3B model. We constructed a dataset of arithmetic expressions with three operands and two operators, varying the order and placement of parentheses. Using this dataset, we trace whether intermediate results appear in the residual stream of the instruction-tuned LLaMA 3.2-3B model. We apply interpretability techniques such as logit lens, linear classification probes, and UMAP geometric visualization. Our results show that intermediate computations are present in the residual stream, particularly after MLP blocks. We also find that the model linearly encodes precedence in each operator's embeddings post attention layer. We introduce partial embedding swap, a technique that modifies operator precedence by exchanging high-impact embedding dimensions between operators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。