arXiv:2411.11278cs.CVcs.MM2024-11CVPR被引 40

让视频事件定位模型识别训练中未见的类别,并给出具体语义标签。

Towards Open-Vocabulary Audio-Visual Event Localization

  • 基于多模态预训练模型构建统一表征空间,融合音视频与文本特征
  • 在24,800段视频上建立新数据集,支持46个已见类与21个未见类的定位
  • 提出无需训练和可微调两种方案,实现开放词汇下的事件识别

音频-视觉事件定位(AVEL)旨在定位并分类同时具有听觉和视觉特征的视频事件。现有研究多在封闭集设置下进行,难以处理训练中未出现的事件类别。本文提出开放词汇音频-视觉事件定位(OV-AVEL)任务,要求模型在推理时对已见和未见事件均能定位并预测具体类别。为此,我们构建了OV-AVEBench数据集,包含24,800段视频,覆盖67个真实场景(已见:未见 = 46:21),每段视频均有逐片段标注。我们还提出了三项评估指标。针对该任务,设计了两种基线方法:一种无训练方案,利用ImageBind的统一多模态空间,通过比较音频-文本与视觉-文本特征相似性的一致性进行预测;另一种采用轻量级时序模块,通过微调方式建模音视频的时序关系。在所提数据集上评估了两种方法,并探讨未来研究方向。

原文摘要 · Abstract (English)

The Audio-Visual Event Localization (AVEL) task aims to temporally locate and classify video events that are both audible and visible. Most research in this field assumes a closed-set setting, which restricts these models' ability to handle test data containing event categories absent (unseen) during training. Recently, a few studies have explored AVEL in an open-set setting, enabling the recognition of unseen events as ``unknown'', but without providing category-specific semantics. In this paper, we advance the field by introducing the Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) problem, which requires localizing audio-visual events and predicting explicit categories for both seen and unseen data at inference. To address this new task, we propose the OV-AVEBench dataset, comprising 24,800 videos across 67 real-life audio-visual scenes (seen:unseen = 46:21), each with manual segment-level annotation. We also establish three evaluation metrics for this task. Moreover, we investigate two baseline approaches, one training-free and one using a further fine-tuning paradigm. Specifically, we utilize the unified multimodal space from the pretrained ImageBind model to extract audio, visual, and textual (event classes) features. The training-free baseline then determines predictions by comparing the consistency of audio-text and visual-text feature similarities. The fine-tuning baseline incorporates lightweight temporal layers to encode temporal relations within the audio and visual modalities, using OV-AVEBench training data for model fine-tuning. We evaluate these baselines on the proposed OV-AVEBench dataset and discuss potential directions for future work in this new field.

多模态开放词汇事件定位音视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。