让多模态大模型自适应地用代码推理，提升算术题解答能力。
AIR: Adaptive Interleaved Reasoning with Code in MLLMs
- 通过强化学习训练模型，实现代码与推理的动态交替。
- 算术推理准确率提升9.9个百分点，工具使用成功率超95%。
- 适合需要精准计算和复杂推理的多模态应用开发人员。
受OpenAI o3启发，将代码融入多模态大语言模型（MLLMs）的交错推理已成为研究前沿。现有方法多聚焦视觉感知任务中的工具调用，依赖预设启发式规则进行视觉操作，难以处理数值计算问题。本文通过在代码增强的复杂数值计算任务上进行扩展强化学习训练，赋予MLLMs自适应交错推理能力。提出三阶段解决方案：两阶段冷启动数据构建流程、用于强化学习数据集筛选的过滤策略，以及基于组约束奖励函数的自适应工具调用策略，以优化交错推理轨迹。大量实验表明，采用组约束奖励函数进行强化学习后，评估基准上性能平均提升6.1个百分点（pp），其中交错推理样本准确率提升9.9 pp，工具使用总体成功率超过95%。相关数据与代码已公开于：https://github.com/CongHan0808/AIR.git。
原文摘要 · Abstract (English)
Following the paradigm shift initiated by OpenAI o3, interleaved reasoning with code to enhance multimodal large language models (MLLMs) has become a pivotal research frontier. The existing literature focuses primarily on tool-use within vision-perception tasks. However, such approaches typically rely on predefined heuristics for visual manipulation and are inherently incapable of addressing numerical computation problems due to their exclusive focus on visual operations. This paper empowers MLLMs with adaptive interleaved reasoning capabilities through extended reinforcement learning training on code-augmented complex numerical computation tasks. To this end, we propose a comprehensive three-component solution consisting of: a two-stage cold-start data construction pipeline, data filtering strategies for RL dataset curation, and an adaptive tool-invocation strategy leveraging a group-constrained reward function for interleaved reasoning trajectories. Extensive experiments demonstrate that after Reinforcement Learning training with the group-constrained reward function, performance improves by an average of 6.1 percentage points (pp) on evaluation benchmarks. Specifically, the accuracy for interleaved reasoning samples increases by 9.9 pp, and the overall success rate of tool-use exceeds 95%. Our data and code are available at: https://github.com/CongHan0808/AIR.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。