提出高效迁移视觉语言模型到开放世界追踪的新方法
EffOWT: Transfer Visual Language Models to Open-World Tracking Efficiently and Effectively
- 在冻结主干网络基础上，添加可学习的轻量侧边网络进行训练
- 未知类别追踪准确率提升5.5%，仅更新1.3%参数，内存减少36.4%
- 适合资源受限场景下需要强泛化能力的开放世界追踪应用
开放世界追踪（OWT）旨在追踪任意类别的所有物体，要求模型具备强大泛化能力。通过利用视觉语言模型（VLM）可提升追踪器的泛化性能。然而，将VLM迁移到OWT时面临挑战：全量微调导致参数与内存开销过大，零样本策略则表现不佳。为此，本文提出EffOWT，通过在VLM主干外构建一个独立的小型可学习侧边网络，冻结主干仅对侧边网络反向传播，实现高效训练。此外，采用融合Transformer与CNN的混合结构增强侧边网络性能，并在MLP中引入稀疏交互机制，显著降低参数更新与内存消耗。实验表明，该方法在未知类别上的跟踪指标OWTA上取得5.5%的绝对提升，相比全量微调仅更新1.3%参数，内存节省达36.4%，其他指标亦有明显改善。
原文摘要 · Abstract (English)
Open-World Tracking (OWT) aims to track every object of any category, which requires the model to have strong generalization capabilities. Trackers can improve their generalization ability by leveraging Visual Language Models (VLMs). However, challenges arise with the fine-tuning strategies when VLMs are transferred to OWT: full fine-tuning results in excessive parameter and memory costs, while the zero-shot strategy leads to sub-optimal performance. To solve the problem, EffOWT is proposed for efficiently transferring VLMs to OWT. Specifically, we build a small and independent learnable side network outside the VLM backbone. By freezing the backbone and only executing backpropagation on the side network, the model's efficiency requirements can be met. In addition, EffOWT enhances the side network by proposing a hybrid structure of Transformer and CNN to improve the model's performance in the OWT field. Finally, we implement sparse interactions on the MLP, thus reducing parameter updates and memory costs significantly. Thanks to the proposed methods, EffOWT achieves an absolute gain of 5.5% on the tracking metric OWTA for unknown categories, while only updating 1.3% of the parameters compared to full fine-tuning, with a 36.4% memory saving. Other metrics also demonstrate obvious improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。