arXiv:2608.00847cs.CV2026-08

用多个模型协作追踪目标,突破单一模型性能瓶颈。

Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Tracking

论文配图:Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Tracking
图 1 · 摘自论文原文
  • 将不同模型当作可调用工具,按事件触发协同工作。
  • 在8个基准上超越最强追踪模型,多模态任务仅用30%参数。
  • 适合需要高精度、长时追踪的视觉系统开发者。

当前视觉追踪器多采用仅在追踪数据集上训练的匹配架构,性能受限于输入上下文长度,已逼近瓶颈。随着高性能追踪依赖基础模型,现有方法将其整体改造为追踪器或分割模型,未能发挥各模型互补优势。基于匹配的追踪器擅长实例对应但缺乏语义区分和精细前景感知,而分割基础模型虽能生成精确掩码,却在实例区分和多模态扩展上表现不佳,且均缺乏长期追踪的纠错能力。为此,我们提出ACTrack,一种代理协调框架,将异构模型视为可调用工具,在事件触发机制下协同工作。ACTrack协调追踪型实例匹配工具(用于目标区分)、SAM3运动工具(基于掩码提供运动先验)、SAM3感知工具(检测干扰物与实例冲突线索),以及仅在持续冲突时激活的VLM重提示工具,以缓解误差累积。我们设计了完整的工具调用触发机制与工具间协调机制,实现不同模型优势的充分融合。实验表明,ACTrack显著超越最强和最大规模追踪器,在8个RGB基准上表现优异。此外,参数高效适配策略支持工具间参数共享复用,仅需30%可训练参数即可实现统一多模态追踪,在LasHeR、VisEvent、TNL2K和DepthTrack等多模态基准上大幅优于先前方法。

原文摘要 · Abstract (English)

Most current visual trackers adopt a matching-based architecture trained exclusively on tracking datasets, whose performance gains depend heavily on the length of the input context, and have now reached a bottleneck. While high-performance tracking increasingly relies on foundation models, existing methods use them monolithically, adapting a foundation model into a tracker or modify a segmentation foundation model into a tracking pipeline, which fails to exploit complementary strengths. Matching-based trackers excel at instance-level correspondence but lack semantic discrimination and fine-grained foreground perception, whereas segmentation foundation models produce precise masks yet struggle with instance discrimination and multimodal extension. Both paradigms also lack error-correction capabilities for long-term tracking. To address these issues, we propose ACTrack, an agentic coordination framework that treats heterogeneous models as invocable tools under an event-triggered mechanism. ACTrack coordinates a Tracker-based Instance Matching Tool for target discrimination, a SAM3 Motion Tool for mask-derived motion priors, a SAM3 Perception Tool for detecting distractors and instance-conflict cues, and a VLM Reprompt Tool activated only under persistent conflict to mitigate error accumulation. We design a complete tool-invocation trigger mechanism and an inter-tool coordination mechanism, enabling the complementary strengths of different model tools to be fully integrated. Experiments show that ACTrack substantially surpasses the strongest and the largest trackers on eight RGB benchmarks. Furthermore, a parameter-efficient adaptation strategy enables parameter sharing and reuse across tools, achieving unified multimodal tracking with only 30\% trainable parameters while substantially outperforming prior methods on multimodal benchmarks such as LasHeR, VisEvent, TNL2K, and DepthTrack.

多模态追踪代理框架模型协作参数效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。