用可执行代码动态组合工具,让AI看图推理更灵活透明。
CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
- 用代码编排多工具,动态计算并可视化中间结果
- 在多个任务上超越闭源模型,且无需微调
- 训练中涌现新工具组合和跨任务迁移能力
近期如 o3 等模型展现了结合工具使用与逐步验证的类人图像推理能力,但多数开源方法仍依赖纯文本链、固定视觉模式或单步流程,限制了复杂任务下的灵活性、可解释性和泛化能力。我们提出 CodeDance,将可执行代码作为通用求解器用于视觉推理。不同于仅预测边界框坐标的固定模式调用,CodeDance 定义、组合并执行代码,以协调多个工具、计算中间结果,并渲染支持透明、自检推理的视觉元素(如框、线、图表)。为引导该过程,我们引入平衡且自适应的工具调用奖励机制,兼顾探索与效率,缓解工具滥用问题。令人意外的是,除原子监督所教能力外,强化学习训练中观察到新型涌现行为:未见的工具调用、全新组合及跨任务迁移。这些行为无需任务特定微调即可出现,表明其具备通用且可扩展的可执行视觉推理潜力。在多个推理基准(如视觉搜索、数学、图表问答)上的大量实验表明,CodeDance 不仅持续优于基于模式驱动和纯文本的基线,还超越 GPT-4o 等闭源模型及更大规模开源模型。
原文摘要 · Abstract (English)
Recent releases such as o3 highlight human-like "thinking with images" reasoning that combines tool use with stepwise verification, yet most open-source approaches still rely on text-only chains, rigid visual schemas, or single-step pipelines, limiting flexibility, interpretability, and transferability on complex tasks. We introduce CodeDance, which explores executable code as a general solver for visual reasoning. Unlike fixed-schema calls (e.g., only predicting bounding-box coordinates), CodeDance defines, composes, and executes code to orchestrate multiple tools, compute intermediate results, and render visual artifacts (e.g., boxes, lines, plots) that support transparent, self-checkable reasoning. To guide this process, we introduce a reward for balanced and adaptive tool calling, which balances exploration with efficiency and mitigates tool overuse. Interestingly, beyond the expected capabilities taught by atomic supervision, we empirically observe novel emergent behaviors during RL training: CodeDance demonstrates novel tool invocations, unseen compositions, and cross-task transfer. These behaviors arise without task-specific fine-tuning, suggesting a general and scalable mechanism for executable visual reasoning. Extensive experiments across reasoning benchmarks (e.g., visual search, math, chart QA) show that CodeDance not only consistently outperforms schema-driven and text-only baselines, but also surpasses closed models such as GPT-4o and larger open-source models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。