arXiv:2608.11292cs.CV2026-08

让AI通过自己写代码来推理图像,实现更可靠的视觉理解。

Self-Evolving Code-with-Image Reasoning

论文配图:Self-Evolving Code-with-Image Reasoning
图 1 · 摘自论文原文
  • 用可执行代码代替语言描述,让模型真正运行视觉算法。
  • 自进化机制使模型从错误代码中学习,准确率从30%提升至67%。
  • 技能可转移,适合研究AI推理与自动化编程的开发者。

多模态模型在解决视觉任务时越来越多地调用工具(如裁剪、缩放、旋转、调亮),这种范式称为‘思考-用图’。核心挑战在于感知:工具主要用于揭示视觉证据,而推理仍停留在语言层面,多数任务人类仅凭观察即可判断。然而,部分视觉问题并不受限于感知,其答案需对像素执行多步视觉算法。对此类问题,模型虽能立刻说出正确算法,但仍会答错,因语言可描述算法却无法执行。代码-图像推理突破此瓶颈:仅提供一个Python解释器,模型必须编写真实视觉算法代码以完成任务,程序本身即为推理过程。此时瓶颈转为选择何种算法。为此我们设计无训练的反思循环:模型分析自身失败代码,测试修复方案并基于正向真值保留有效策略,形成可复用的技能。在我们构建的代码-图像基准(CwI-Bench)上,30个任务族由隐藏视觉计算生成,且训练与评估集互不重叠;即使使用GPT-5.6-luna,纯链式思维工具调用准确率低于30%,仅给解释器则达43%,经自我可执行反思进化后提升至67%。开放27B模型也实现相同跃升(9% → 33% → 56%),且技能为纯文本,可在不同规模和任务族间迁移。当代码承载推理,调试代码即等价于调试推理。

原文摘要 · Abstract (English)

Multimodal models increasingly reach for tools when solving visual tasks (crop, zoom, rotate, brighten), a paradigm known as thinking-with-images. The central challenge is one of perception: tools mostly serve to expose visual evidence, reasoning over that evidence stays in language, and most targets are ones a human could in principle determine by inspection. Some visual questions, however, are not bottlenecked by perception: recovering their answers requires executing a multi-step visual algorithm over the pixels. On such questions a model often names the correct algorithm at once yet still answers wrong, because language can describe an algorithm without being able to run one. Code-with-Image crosses that line: given nothing but a Python interpreter, the model must implement a genuine visual algorithm in code to solve the task; the program itself becomes the reasoning. The bottleneck then shifts from executing code to deciding which algorithm to implement. So we let the model teach itself: a training-free reflection loop studies its own failed programs, tests repairs against constructive ground truth, and keeps what survives as portable skills. On our Code-with-Image Bench (CwI-Bench), thirty task families induced by hidden visual computations with disjoint learning and evaluation splits, even GPT-5.6-luna stays below 30% with tool-free chain of thought; given a bare interpreter it reaches 43%, and with skills evolved through its own executable reflection, 67%. The open 27B model climbs the same ladder (9% $\rightarrow$ 33% $\rightarrow$ 56%), and the skills are plain text, transferable across scales and families. When code carries the reasoning, debugging code becomes debugging reasoning.

代码推理视觉算法自进化多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。