让数学大模型自主决定何时用代码，提升解题能力。
To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization
- 用期望最大化框架动态学习代码使用时机。
- 7B模型在MATH500上提升11%，AIME上提升9.4%。
- 适合想让模型自适应调用工具的研究者。
近期数学问题求解的大型语言模型结合思维链（CoT）推理与代码执行，发挥互补优势。但现有混合框架依赖外部指令或固定模板，缺乏元认知能力——即动态评估自身能力并自主决定工具使用时机与方式。这种僵化限制了模型在训练中随着推理能力演进而调整策略的能力。尽管强化学习（RL）在大规模提升大模型推理方面有潜力（如DeepSeek-R1），但其在学习自主代码集成时效率低下，因难以探索思维链与代码交织模式的庞大组合空间。为此，我们提出一种新的期望-最大化（EM）框架，通过结构化探索（E步）与离策略强化学习优化（M步）协同，形成元认知决策与能力演进的自我强化循环。实验表明，该方法显著提升探索效率，取得更优结果：7B模型在MATH500上提升超过11%，在AIME上提升9.4%，且无需类似o1的思维链提示。
原文摘要 · Abstract (English)
Recent advances in mathematical problem-solving with language models (LMs) integrate chain-of-thought (CoT) reasoning and code execution to harness their complementary strengths. However, existing hybrid frameworks exhibit a critical limitation: they depend on externally dictated instructions or rigid code-integration templates, lacking metacognitive awareness -- the capacity to dynamically evaluate intrinsic capabilities and autonomously determine when and how to integrate tools. This rigidity motivates our study of autonomous code integration, enabling models to adapt tool-usage strategies as their reasoning abilities evolve during training. While reinforcement learning (RL) shows promise for boosting LLM reasoning at scale (e.g., DeepSeek-R1), we demonstrate its inefficiency in learning autonomous code integration due to inadequate exploration of the vast combinatorial space of CoT-code interleaving patterns. To address this challenge, we propose a novel Expectation-Maximization (EM) framework that synergizes structured exploration (E-step) with off-policy RL optimization (M-step), creating a self-reinforcing cycle between metacognitive tool-use decisions and evolving capabilities. Experiments reveal our method achieves superior results through improved exploration. Notably, our 7B model improves over 11% on MATH500 and 9.4% on AIME without o1-like CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。