arXiv:2505.17114cs.CLcs.CV2025-05EMNLP被引 3

让多模态问答更准:用问题引导注意力,自动过滤干扰信号。

RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language

  • 根据问题动态筛选音视频与传感器数据中的关键片段
  • 在7个基准上比现有模型高14.5%,加传感器再提16.4%
  • 适合做智能助手、自动驾驶等需融合多源信息的场景

多模态问答常需识别音视频或传感器中与问题相关的内容。但不同模态间常存在不一致:如镜头外说话、背景噪音或视域外运动,会误导均等加权的融合模型。本文提出RAVEN,核心是QuART——一种查询条件化的跨模态门控模块,为各模态的每个标记分配相关性得分,使模型在融合前增强有效信号、抑制干扰。RAVEN采用三阶段训练流程:单模态预训练、查询对齐融合、分歧导向微调,分别应对表征质量、跨模态相关性与模态不匹配的挑战。为此我们发布了包含30万条同步音视频传感器流及自动生成问答对的AVS-QA数据集。在7个多模态问答基准(含第一人称与第三人称任务)上的实验表明,相比当前最优多模态大模型,RAVEN准确率提升最高达14.5%和8.0%;加入传感器数据额外提升16.4%,在模态损坏情况下仍保持鲁棒,优于最先进基线50.23%。代码与数据集见https://github.com/BASHLab/RAVEN。

原文摘要 · Abstract (English)

Multimodal question answering (QA) often requires identifying which video, audio, or sensor tokens are relevant to the question. Yet modality disagreements are common: off-camera speech, background noise, or motion outside the field of view often mislead fusion models that weight all streams equally. We present RAVEN, a unified QA architecture whose core is QuART, a query-conditioned cross-modal gating module that assigns scalar relevance scores to each token across modalities, enabling the model to amplify informative signals and suppress distractors before fusion. RAVEN is trained through a three-stage pipeline comprising unimodal pretraining, query-aligned fusion, and disagreement-oriented fine-tuning -- each stage targeting a distinct challenge in multi-modal reasoning: representation quality, cross-modal relevance, and robustness to modality mismatch. To support training and evaluation, we release AVS-QA, a dataset of 300K synchronized Audio--Video-Sensor streams paired with automatically generated question-answer pairs. Experimental results on seven multi-modal QA benchmarks -- including egocentric and exocentric tasks -- show that RAVEN achieves up to 14.5\% and 8.0\% gains in accuracy compared to state-of-the-art multi-modal large language models, respectively. Incorporating sensor data provides an additional 16.4\% boost, and the model remains robust under modality corruption, outperforming SOTA baselines by 50.23\%. Our code and dataset are available at https://github.com/BASHLab/RAVEN.

多模态问答音视频分析传感器融合注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。