让视觉大模型自主编写代码工具，解决复杂视觉任务
PyVision: Agentic Vision with Dynamic Tooling
- 模型可动态生成、执行和优化自定义Python工具
- 在V*上提升GPT-4.1性能7.8%，在VLMsAreBlind-mini上提升Claude-4.0-Sonnet 31.1%
- 适合需要灵活视觉推理的AI研发人员
大语言模型正被用于构建具备规划、推理和动态调用外部工具能力的智能体。然而，在视觉推理领域，现有方法仍受限于预设流程和静态工具集。本文提出PyVision，一个支持多轮交互的框架，使多模态大模型能自主生成、执行并优化针对具体任务的Python工具，实现灵活且可解释的问题求解。我们构建了PyVision生成工具的分类体系，并分析其在多样化基准上的使用情况。定量结果显示，PyVision在V*上使GPT-4.1性能提升7.8%，在VLMsAreBlind-mini上使Claude-4.0-Sonnet性能提升31.1%。这些结果表明：动态工具化使模型不仅能使用工具，还能创造工具，推动视觉推理向更智能的代理方向演进。
原文摘要 · Abstract (English)
LLMs are increasingly deployed as agents, systems capable of planning, reasoning, and dynamically calling external tools. However, in visual reasoning, prior approaches largely remain limited by predefined workflows and static toolsets. In this report, we present PyVision, an interactive, multi-turn framework that enables MLLMs to autonomously generate, execute, and refine Python-based tools tailored to the task at hand, unlocking flexible and interpretable problem-solving. We develop a taxonomy of the tools created by PyVision and analyze their usage across a diverse set of benchmarks. Quantitatively, PyVision achieves consistent performance gains, boosting GPT-4.1 by +7.8% on V* and Claude-4.0-Sonnet by +31.1% on VLMsAreBlind-mini. These results point to a broader shift: dynamic tooling allows models not just to use tools, but to invent them, advancing toward more agentic visual reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。