让大模型用代码操控图像,提升视觉推理的鲁棒性和灵活性。
Thinking with Programming Vision: Towards a Unified View for Thinking with Images
- 用生成代码代替固定工具,实现任意图像操作的通用接口。
- 在旋转和自然退化图像上性能提升显著,错误恢复能力增强。
- 适合需要复杂图像处理与可靠推理的AI研究者与开发者。
多模态大语言模型虽能通过工具进行图像推理,但现有方法依赖有限工具,缺乏真实场景下的必要性与可扩展性。本文揭示当前顶级模型在简单视角变化或自然退化图像上表现脆弱,亟需更鲁棒的工具化推理机制。为此提出CodeVision框架,以代码作为通用接口调用任意图像操作,突破固定工具库限制。采用两阶段训练:先在高质量数据集上进行监督微调(SFT),覆盖复杂多轮工具组合与错误恢复;再通过强化学习(RL)结合新型密集过程奖励函数,鼓励策略性高效执行。构建了新的SFT与RL数据集,并引入挑战性新基准,严格评估对视角变换和多工具推理的鲁棒性。在Qwen2.5-VL与Qwen3-VL系列模型上实验表明,该方法显著提升性能,催生灵活工具组合、高效链式执行及基于运行时反馈的鲁棒纠错等涌现能力。代码已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) that think with images can interactively use tools to reason about visual inputs, but current approaches often rely on a narrow set of tools with limited real-world necessity and scalability. In this work, we first reveal a critical and previously overlooked weakness: even state-of-the-art MLLMs are surprisingly brittle, showing significant performance degradation on images with simple orientation changes or natural corruptions, underscoring the need for more robust tool-based reasoning. To address this, we propose CodeVision, a flexible and scalable code-as-tool framework where the model generates code as a universal interface to invoke any image operation, moving beyond fixed tool registries. We train our model using a two-stage methodology, beginning with Supervised Fine-Tuning (SFT) on a high-quality dataset curated for complex, multi-turn tool composition and error recovery, followed by Reinforcement Learning (RL) with a novel and dense process reward function to encourage strategic and efficient tool use. To facilitate this research, we construct new SFT and RL datasets and introduce a challenging new benchmark suite designed to rigorously evaluate robustness to orientation changes and multi-tool reasoning. Experiments on Qwen2.5-VL and Qwen3-VL series show that our approach significantly improves model performance and fosters emergent capabilities such as flexible tool composition, efficient chained execution, and robust error recovery from runtime feedback. Code is available at https://github.com/ByteDance-BandAI/CodeVision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。