用语言提示提升高光谱目标跟踪精度,解决冗余与形变难题
Vision-Language Guided Hyperspectral Object Tracking via Semantics Fusion and Contextual Template Updating

- 通过语言引导选谱模块减少高光谱冗余,突出关键波段特征
- 融合视觉与语言信息,实现跨模态表征学习,提升追踪鲁棒性
- 动态更新模板特征,适应长序列中目标形变,适合复杂场景追踪
高光谱目标跟踪(HOT)利用高光谱视频(HSVs)丰富的光谱信息,在目标追踪领域具有巨大潜力。然而,如何高效提取并利用冗余波段中的光谱信息仍是核心挑战,严重制约模型泛化能力与追踪性能。此外,在动态场景中,目标常因遮挡、光照变化等导致外观剧烈变化,造成当前帧与模板间显著形变,给现有时序建模方法带来重大挑战。本文提出VLHTrack,一种新颖的高光谱视觉-语言联合追踪框架。首先,设计语言引导波段选择模块(LBSM),借助大语言模型(LLM)描述建立语义到光谱的映射,缓解冗余并强化判别性特征。其次,引入多模态视觉-语言融合模块,无缝集成视觉与语言嵌入,利用其互补优势学习一致的跨模态表示。为应对长期序列中的目标形变,提出基于马比(Mamba)的动态模板更新模块(DTUM),通过选择性状态空间建模学习帧间依赖关系,实现受时序上下文指导的高效模板特征演化。在HOT2023和HOT2024数据集上的实验表明,VLHTrack优于现有最先进方法。
原文摘要 · Abstract (English)
Hyperspectral object tracking (HOT) leverages the rich spectral information provided by hyperspectral videos (HSVs), offering substantial potential for object tracking. However, efficiently extracting and exploiting spectral information from redundant spectral bands remains a fundamental challenge, which severely limits model generalization and tracking performance. Moreover, in dynamic scenes, targets often experience drastic appearance variations due to factors such as occlusion and illumination changes. These variations lead to large deformations between the current frame and the template. Such discrepancies pose major challenges for existing temporal modeling approaches. In this work, we propose VLHTrack, a novel hyperspectral vision-language (VL) joint tracking framework. Specifically, we incorporate language priors to address the fundamental challenge of spectral redundancy by designing a Language-Guided Band Selection Module (LBSM). By leveraging Large Language Model (LLM) descriptions, LBSM establishes a semantic-to-spectral mapping that mitigates redundancy and accentuates discriminative spectral features. A Multi-Modal Vision-Language Fusion Module is then employed to seamlessly integrate visual and linguistic embeddings, harnessing their complementary advantages to learn coherent cross-modal representations. To address target deformation in long-term sequences, we propose a dynamic update template feature strategy implemented via the Dynamic Template Update with Mamba (DTUM) module. By leveraging selective state space modeling, DTUM learns inter-frame dependencies to update template feature, ensuring efficient template feature evolution guided by temporal context. Experiments on HOT2023 and HOT2024 demonstrate that VLHTrack outperforms state-of-the-art (SOTA) methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。