提出轻量级追踪框架,跨模态高效建模目标动态与历史上下文。
Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modality
- 用压缩提示词和动态状态融合,解耦记忆与运动变化建模。
- 仅训练30%参数,在10个数据集上达顶尖性能。
- 可即插即用,适配多模态追踪任务的快速部署需求。
随着基于Transformer的一流追踪器在帧间关系建模方面表现强劲,研究日益关注如何引入时空上下文。然而,现有方法通常依赖有限的历史帧,导致上下文利用不足且输入过长,计算开销巨大;而通过外部记忆库查询的方法又存在检索特征与主干网络融合不充分的问题。此外,以离散历史帧作为上下文忽略了目标的连续动态特性。为此,我们提出Uni-MDTrack,包含两个核心模块:记忆感知压缩提示(MCP)模块与动态状态融合(DSF)模块。MCP将丰富的记忆特征压缩为记忆感知提示令牌,与输入在整个主干中深度交互,显著提升性能并保持稳定计算负载。DSF通过从浅层到深层逐步引入更新的动态状态特征,补充离散记忆,同时维持高效率。Uni-MDTrack支持跨RGB、RGB-D/T/E及RGB-语言模态的统一追踪。实验表明,仅训练MCP、DSF和预测头,可使可训练参数比例保持在约30%,在覆盖五种模态的10个数据集上实现显著性能提升,并达到当前最佳水平。此外,两个模块均展现出优异泛化性,可作为即插即用组件提升多种基线追踪器性能,显著优于现有参数高效训练方法。
原文摘要 · Abstract (English)
With the advent of Transformer-based one-stream trackers that possess strong capability in inter-frame relation modeling, recent research has increasingly focused on how to introduce spatio-temporal context. However, most existing methods rely on a limited number of historical frames, which not only leads to insufficient utilization of the context, but also inevitably increases the length of input and incurs prohibitive computational overhead. Methods that query an external memory bank, on the other hand, suffer from inadequate fusion between the retrieved spatio-temporal features and the backbone. Moreover, using discrete historical frames as context overlooks the rich dynamics of the target. To address the issues, we propose Uni-MDTrack, which consists of two core components: Memory-Aware Compression Prompt (MCP) module and Dynamic State Fusion (DSF) module. MCP effectively compresses rich memory features into memory-aware prompt tokens, which deeply interact with the input throughout the entire backbone, significantly enhancing the performance while maintaining a stable computational load. DSF complements the discrete memory by capturing the continuous dynamic, progressively introducing the updated dynamic state features from shallow to deep layers, while also preserving high efficiency. Uni-MDTrack also supports unified tracking across RGB, RGB-D/T/E, and RGB-Language modalities. Experiments show that in Uni-MDTrack, training only the MCP, DSF, and prediction head, keeping the proportion of trainable parameters around 30%, yields substantial performance gains, achieves state-of-the-art results on 10 datasets spanning five modalities. Furthermore, both MCP and DSF exhibit excellent generality, functioning as plug-and-play components that can boost the performance of various baseline trackers, while significantly outperforming existing parameter-efficient training approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。