无需微调,代码增强让普通语言模型推理能力超越专用推理模型。
Code-enabled language models can outperform reasoning models on diverse tasks
- 用代码执行+少样本提示,让普通模型学会多步推理。
- 在8个任务上平均超越对应推理模型,最高提升22.9%。
- 适合想低成本提升模型推理能力的研究者与开发者。
推理模型(RMs)虽成功但需大量算力和数据训练,且运行成本高。本文发现,未经微调的标准指令型语言模型(LMs),通过CodeAdapt方法——结合CodeAct框架(模型交替使用自然语言与代码执行)和仅5个示例的少样本上下文学习——可在指令遵循、创意生成、数学推理等多样任务上达到甚至超过对应推理模型的表现(如DeepSeek V3 vs R1)。分析4对匹配的LM与RM,在8个任务上3个LM平均超越对应RM(最高提升22.9%),且令牌效率高出10%-81%;在4个模型平均下,6个任务表现更优(最高提升35.7%)。代码增强的推理过程展现出丰富多样的解题策略。结果表明:(1) CodeAdapt式学习具备强鲁棒性和跨领域泛化能力;(2) 代码赋能的LM具有认知根基,是强大的系统,或可作为权内强化学习的坚实基础。
原文摘要 · Abstract (English)
Reasoning models (RMs), language models (LMs) trained with reinforcement learning to produce long-form natural language reasoning, have been remarkably successful, but they still require large amounts of computation and data to train, and can be slow and expensive to run. In this paper, we show that standard instruct LMs can already be elicited to be strong reasoners at a level comparable to or even surpassing their corresponding RMs (e.g., DeepSeek V3 vs R1) without finetuning, across diverse domains from instruction following and creative generation to mathematical reasoning. This is achieved by CodeAdapt, our simple recipe that combines the CodeAct framework, where LMs interleave natural language reasoning with code execution in a multi-step fashion, with few-shot bootstrap in-context learning from as few as five training problems. Analyzing four matched pairs of LMs and RMs, we find that CodeAdapt enables three LMs to outperform the corresponding RMs on average over eight tasks (up to 22.9%) while being 10-81% more token efficient, and delivers superior performance on six tasks when averaged over the four models (up to 35.7%). Furthermore, the code-augmented reasoning traces display rich and varied problem-solving strategies. Our findings support that (1) CodeAdapt-style learning and reasoning may be robust and domain general and (2) code-enabled LMs are cognitively grounded and powerful systems, potentially providing a strong foundation for in-weight reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。