arXiv:2606.20728cs.CVcs.CL2026-06

让视觉工具自动搭配顺序与参数,应对复杂场景下的识别难题。

VTOS: Learning to Orchestrate Vision Tools by Co-Searching Solutions and Observers

论文配图:VTOS: Learning to Orchestrate Vision Tools by Co-Searching Solutions and Observers
图 1 · 摘自论文原文
  • 联合搜索解决方案与诊断程序,动态调整工具使用策略。
  • 在LVIS-Count和PlantSeg-OOD上显著优于固定流程与现有智能体。
  • 适合需要自适应视觉工具链的复杂视觉任务开发者使用。

视觉基础工具如开集检测器、分割模型和后处理算子是计算机视觉的重要构建模块,但其效果高度依赖于工具的编排方式:使用哪些工具、以何种顺序、参数如何设置、在何种视觉条件下运行。现有视觉编程智能体通常生成固定解决方案流程,在密集物体、遮挡、小目标和域偏移等情况下表现脆弱。我们提出VTOS(视觉工具编排搜索)框架,通过联合搜索解决方案与观察者程序实现自适应编排。VTOS协同搜索由Grounding DINO、SAM、NMS、切片-检测等组成的可执行解决方案程序,以及能诊断候选方案、识别失败模式并生成可操作反馈的观察者程序。这些观测结果被存入共享的VisionThoughts知识库,指导后续搜索。我们在两个案例研究中评估VTOS:LVIS-Count上的密集物体计数与PlantSeg-OOD上的零样本植物病害分割,分别挑战阈值校准、NMS、切片、掩码优化和领域泛化等编排问题。在两项任务中,VTOS均超越静态工具流程和代理式视觉编程基线,证明联合搜索解决方案与观察者是应对复杂视觉任务的有效策略。

原文摘要 · Abstract (English)

Vision foundation tools such as open-vocabulary detectors, segmentation models, and post-processing operators are powerful building blocks for computer vision, but their effectiveness depends heavily on how they are orchestrated: which tools are used, in what order, with what parameters, and under what visual conditions. Existing visual-programming agents typically generate a fixed solution pipeline, making them brittle under dense objects, occlusion, small targets, and domain shift. We introduce VTOS (Vision Tools Orchestration Search), a framework for adaptive visual tool orchestration through joint solution--observer search. VTOS co-searches executable solution programs that compose vision tools such as Grounding DINO, SAM, NMS, and slice-and-detect, together with observer programs that diagnose candidate solutions, identify failure modes, and generate actionable feedback. These observations are accumulated in a shared VisionThoughts knowledge base to guide subsequent search. We evaluate VTOS through two case studies: dense object counting on LVIS-Count and zero-shot plant-disease segmentation on PlantSeg-OOD, which stress different orchestration challenges including threshold calibration, NMS, slicing, mask refinement, and domain generalization. Across both tasks, VTOS outperforms static tool pipelines and agentic visual-programming baselines, showing that co-searching solutions and observers is an effective strategy for adapting vision tools to challenging computer vision tasks.

工具编排视觉智能体自适应推理零样本分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。