LLM解数学应用题时，算术计算比抽象建模更拖后腿。
Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation
- 通过分离评估抽象建模与算术计算，发现算术是主要瓶颈。
- 无思维链的Llama-3和Qwen2.5在算术上表现远差于建模能力。
- 揭示模型在单次前向传播中已具备抽象-计算机制，适合评估改进方向。
大语言模型在数学应用题上的最终答案准确率常被用作推理能力的代理指标。然而，该指标混淆了两个不同子技能：抽象建模（用表达式捕捉数学关系）和算术计算（执行具体运算）。在GSM8K和SVAMP数据集上进行解耦评估后发现，无思维链的Llama-3和Qwen2.5（1B-32B）的最终答案准确率主要受限于算术计算步骤，而非抽象建模。与普遍认知相反，我们证明思维链主要帮助算术计算，对抽象建模影响有限。机制分析显示，这两个技能在单次前向传播中以‘先抽象、再计算’的方式协同实现；因果修补验证了这些抽象表征的存在性、可迁移性、可组合性，并先于计算发生。行为与机制结果共同强调了解耦评估的重要性，以准确衡量大模型推理能力并指导未来优化。
原文摘要 · Abstract (English)
Final-answer-based metrics are commonly used for evaluating large language models (LLMs) on math word problems, often taken as proxies for reasoning ability. However, such metrics conflate two distinct sub-skills: abstract formulation (capturing mathematical relationships using expressions) and arithmetic computation (executing the calculations). Through a disentangled evaluation on GSM8K and SVAMP, we find that the final-answer accuracy of Llama-3 and Qwen2.5 (1B-32B) without CoT is overwhelmingly bottlenecked by the arithmetic computation step and not by the abstract formulation step. Contrary to the common belief, we show that CoT primarily aids in computation, with limited impact on abstract formulation. Mechanistically, we show that these two skills are composed conjunctively even in a single forward pass without any reasoning steps via an abstract-then-compute mechanism: models first capture problem abstractions, then handle computation. Causal patching confirms these abstractions are present, transferable, composable, and precede computation. These behavioural and mechanistic findings highlight the need for disentangled evaluation to accurately assess LLM reasoning and to guide future improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。