利用音频生成伪标签,提升视频模型测试时自适应性能。
Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation
- 用预训练音频模型分类音轨,通过大模型映射到视频标签空间。
- 在多个数据集上,显著优于仅用视觉信号的现有方法。
- 适合关注多模态自适应、视频分类模型部署的开发者。
测试时自适应(TTA)旨在通过测试阶段的自/无监督学习提升已训练模型的泛化能力。现有视频TTA方法主要依赖视觉信号,常忽略音频数据的潜在价值。为此,本文提出一种将音频信息融入视频TTA的新方法,利用音频丰富的语义内容生成音频辅助伪标签。具体地,先使用预训练音频模型对视频中提取的音频信号进行分类,再通过大语言模型将音频预测映射至视频标签空间,建立音频类别与视频标签的关联。为有效利用生成的伪标签,设计了一种灵活的自适应循环机制,根据损失变化和不同视图间一致性动态决定每样本的最佳适应迭代次数,实现样本级定制化适配。在两个常用数据集(UCF101-C 和 Kinetics-Sounds-C),以及两个新构建的音视频TTA数据集(AVE-C 和 AVMIT-C)上,面对多种扰动类型,实验结果均显示本方法在不同视频分类模型上持续提升适配性能,标志着音视频融合在视频TTA中的重要进展。代码开源:https://github.com/keikeiqi/Audio-Assisted-TTA。
原文摘要 · Abstract (English)
Test-time adaptation (TTA) aims to boost the generalization capability of a trained model by conducting self-/unsupervised learning during the testing phase. While most existing TTA methods for video primarily utilize visual supervisory signals, they often overlook the potential contribution of inherent audio data. To address this gap, we propose a novel approach that incorporates audio information into video TTA. Our method capitalizes on the rich semantic content of audio to generate audio-assisted pseudo-labels, a new concept in the context of video TTA. Specifically, we propose an audio-to-video label mapping method by first employing pre-trained audio models to classify audio signals extracted from videos and then mapping the audio-based predictions to video label spaces through large language models, thereby establishing a connection between the audio categories and video labels. To effectively leverage the generated pseudo-labels, we present a flexible adaptation cycle that determines the optimal number of adaptation iterations for each sample, based on changes in loss and consistency across different views. This enables a customized adaptation process for each sample. Experimental results on two widely used datasets (UCF101-C and Kinetics-Sounds-C), as well as on two newly constructed audio-video TTA datasets (AVE-C and AVMIT-C) with various corruption types, demonstrate the superiority of our approach. Our method consistently improves adaptation performance across different video classification models and represents a significant step forward in integrating audio information into video TTA. Code: https://github.com/keikeiqi/Audio-Assisted-TTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。