提出轻量级适配器,让旧模型高效追踪高光谱视频中的目标。
Hyperspectral Adapter for Object Tracking based on Hyperspectral Video
- 用两个适配器在自注意力和MLP中注入高光谱信息
- 仅微调新模块,参数量少于传统方法的1/10
- 在4个数据集上均达顶尖性能,适合实时应用
基于高光谱视频的目标跟踪因蕴含丰富的材质与运动信息而受到关注。现有方法通过在高光谱数据集上微调预训练的RGB追踪网络,虽取得良好效果,但存在光谱信息在转换过程中的损失,且全网微调效率低下。为此,本文提出高光谱适配器追踪方法(HyA-T),包括用于自注意力的高光谱适配器(HAS)和用于多层感知机的高光谱适配器(HAM),通过将适配信息注入多头自注意力(MSA)和多层感知机(MLP)的计算过程,实现对预训练网络的高效适配。同时,提出高光谱输入增强(HEI)模块,将原始光谱信息直接融入网络输入。该方法直接从高光谱图像提取光谱特征,避免信息丢失。仅需微调新增模块的参数,显著提升效率。在包含不同光谱波段的四个数据集上进行的大量实验验证了其有效性,所提方法在所有数据集上均达到当前最优性能。
原文摘要 · Abstract (English)
Object tracking based on hyperspectral video attracts increasing attention to the rich material and motion information in the hyperspectral videos. The prevailing hyperspectral methods adapt pretrained RGB-based object tracking networks for hyperspectral tasks by fine-tuning the entire network on hyperspectral datasets, which achieves impressive results in challenging scenarios. However, the performance of hyperspectral trackers is limited by the loss of spectral information during the transformation, and fine-tuning the entire pretrained network is inefficient for practical applications. To address the issues, a new hyperspectral object tracking method, hyperspectral adapter for tracking (HyA-T), is proposed in this work. The hyperspectral adapter for the self-attention (HAS) and the hyperspectral adapter for the multilayer perceptron (HAM) are proposed to generate the adaption information and to transfer the multi-head self-attention (MSA) module and the multilayer perceptron (MLP) in pretrained network for the hyperspectral object tracking task by augmenting the adaption information into the calculation of the MSA and MLP. Additionally, the hyperspectral enhancement of input (HEI) is proposed to augment the original spectral information into the input of the tracking network. The proposed methods extract spectral information directly from the hyperspectral images, which prevent the loss of the spectral information. Moreover, only the parameters in the proposed methods are fine-tuned, which is more efficient than the existing methods. Extensive experiments were conducted on four datasets with various spectral bands, verifing the effectiveness of the proposed methods. The HyA-T achieves state-of-the-art performance on all the datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。