让AI通过经验自动生成空间推理工具,性能比GPT-4o高22%。
Transductive Visual Programming: Evolving Tool Libraries from Experience for Spatial Reasoning
- 用实际解题经验积累程序,从中提炼可复用的高级工具。
- 在Omni3D-Bench上超越GPT-4o 22%,工具使用频率提升5倍。
- 适合需要自主进化能力的空间推理任务研究者。
3D场景中的空间推理需要精确几何计算,挑战视觉语言模型能力。视觉编程通过分步调用专用工具解决此问题,但现有方法依赖固定工具集或解题前推测工具,导致程序效果差且工具利用率低。本文提出转导式视觉编程(TVP),不靠推测而从自身经验构建新工具。TVP先用基础工具解题,并将解题过程存入示例库;再从这些程序中抽象出重复模式,形成可复用的高层工具,构建持续进化的工具库。这使TVP能用不断演化的工具应对新问题。在Omni3D-Bench上,TVP表现达当前最优,比GPT-4o高22%,比此前最佳视觉编程系统高11%。其转导学习工具作为核心依赖的使用频率是归纳生成工具的5倍,体现更优的工具发现与复用能力。演化工具还展现出强泛化性,在SpatialScore-Hard未见任务上无需修改即取得优异表现。本工作确立了以经验驱动的转导工具创建为强大范式,推动自演化视觉编程代理有效应对复杂空间推理任务。代码已公开于https://transductive-visualprogram.github.io/。
原文摘要 · Abstract (English)
Spatial reasoning in 3D scenes requires precise geometric calculations that challenge vision-language models. Visual programming addresses this by decomposing problems into steps calling specialized tools, yet existing methods rely on either fixed toolsets or speculative tool induction before solving problems, resulting in suboptimal programs and poor utilization of induced tools. We present Transductive Visual Programming (TVP), a novel framework that builds new tools from its own experience rather than speculation. TVP first solves problems using basic tools while accumulating experiential solutions into an Example Library, then abstracts recurring patterns from these programs into reusable higher-level tools for an evolving Tool Library. This allows TVP to tackle new problems with increasingly powerful tools learned from experience. On Omni3D-Bench, TVP achieves state-of-the-art performance, outperforming GPT-4o by 22% and the previous best visual programming system by 11%. Our transductively learned tools are used 5x more frequently as core program dependency than inductively created ones, demonstrating more effective tool discovery and reuse. The evolved tools also show strong generalization to unseen spatial tasks, achieving superior performance on benchmarks from SpatialScore-Hard collection without any testset-specific modification. Our work establishes experience-driven transductive tool creation as a powerful paradigm for building self-evolving visual programming agents that effectively tackle challenging spatial reasoning tasks. We release our code at https://transductive-visualprogram.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。