让智能体学会调用专家模型,解决复杂视觉任务
Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning

- 训练智能体而非专家,通过多轮对话调用不同视觉专家
- 在4个基准上超越通用模型,接近专用模型性能
- 适合需要跨任务推理的复杂视觉应用
近期计算机视觉进展催生了众多针对检测、分割、计数等任务的强大专用模型。然而这些模型通常针对孤立任务优化,难以直接支持通用视觉智能,尤其在需要复杂语言理解与密集小物体感知的任务中。本文提出 VisHarness,一种可训练的视觉智能体,将高层感知、推理与决策与底层任务执行解耦。不训练模型直接解决特定视觉任务,而是让 VisHarness 学习如何调用一组精心设计的异构视觉专家。该范式在保持智能体通用性的同时,充分借助专用模型在具体任务中的精度优势。仅需轻量级训练,VisHarness 即可学习通用的专家调用策略,在多轮交互下应对多种复杂条件完成基础视觉任务。为实现高效在线强化学习,引入动态视觉记忆归档机制,缓解多轮交互带来的视觉标记积压问题。在涵盖推理分割、广义指代分割、密集小物体检测和指代计数的四个代表性基准上,VisHarness 显著优于现有通用模型,并达到或超越特定任务模型的性能。
原文摘要 · Abstract (English)
Recent progress in computer vision has produced a wide range of powerful specialized models for detection, segmentation, counting, and other visual tasks. However, these models are usually optimized for isolated task formulations, making it difficult to directly support general-purpose visual intelligence, especially when a task requires complex language understanding and dense small-object perception. In this paper, we propose VisHarness, a trainable visual agent that decouples high-level perception, reasoning, and decision-making from low-level task execution. Instead of training a model to solve a specific visual task, VisHarness learns to harness a set of carefully designed heterogeneous visual experts. This paradigm preserves the general intelligence of the agent while fully leveraging the precision advantages of specialized visual models in concrete visual tasks. With only lightweight training, VisHarness learns a generalizable visual expert-harnessing policy and can solve common fundamental vision tasks under various complex conditions through multi-turn interactions with visual expert models. To enable efficient on-policy reinforcement learning training in a live environment, we introduce dynamic visual memory archiving, which mitigates the rapidly accumulating visual-token overhead caused by multi-turn interactions with visual expert models. Experiments on four representative benchmarks covering reasoning segmentation, generalized referring segmentation, dense small-object detection, and referring counting demonstrate that VisHarness substantially outperforms existing general-purpose models and achieves competitive or superior performance compared with task-specific models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。