用专家混合模型提升视觉追踪的时空关系建模能力,参数高效且精度领先。
SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking
- 引入专用于追踪任务的专家混合架构,灵活处理不同区域的关系建模需求
- 在7个数据集上显著超越现有最优方法,参数增长极小
- 适合作为可扩展追踪系统的核心模块,尤其适合资源受限场景
当前主流追踪器多采用单流Vision Transformer架构,联合提取模板与搜索区域特征并建模其关系。然而,图像块间关系存在显著差异:背景区域含大量无关信息,应降低注意力;而前景尤其是边界区域则需重点关注。单一模型难以兼顾所有情形。本文提出新型追踪器SPMTrack,基于专为追踪设计的专家混合架构(TMoE),通过多个专家协同实现更灵活的关系建模。得益于TMoE,我们将关系建模拓展至时空上下文,显著提升追踪精度,同时模型参数几乎不变。此外,将TMoE作为参数高效微调方法,大幅减少可训练参数,支持不同规模模型高效训练,并保持预训练模型泛化能力。在七个数据集上的实验表明,本方法显著优于现有最优追踪器。代码已开源。
原文摘要 · Abstract (English)
Most state-of-the-art trackers adopt one-stream paradigm, using a single Vision Transformer for joint feature extraction and relation modeling of template and search region images. However, relation modeling between different image patches exhibits significant variations. For instance, background regions dominated by target-irrelevant information require reduced attention allocation, while foreground, particularly boundary areas, need to be be emphasized. A single model may not effectively handle all kinds of relation modeling simultaneously. In this paper, we propose a novel tracker called SPMTrack based on mixture-of-experts tailored for visual tracking task (TMoE), combining the capability of multiple experts to handle diverse relation modeling more flexibly. Benefiting from TMoE, we extend relation modeling from image pairs to spatio-temporal context, further improving tracking accuracy with minimal increase in model parameters. Moreover, we employ TMoE as a parameter-efficient fine-tuning method, substantially reducing trainable parameters, which enables us to train SPMTrack of varying scales efficiently and preserve the generalization ability of pretrained models to achieve superior performance. We conduct experiments on seven datasets, and experimental results demonstrate that our method significantly outperforms current state-of-the-art trackers. The source code is available at https://github.com/WenRuiCai/SPMTrack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。