用多智能体框架解决跨视频找关键片段的复杂推理问题
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
- 设计多智能体系统，协同分析多视频中的音视频线索
- 在3100对标注数据上，比基线提升89%和65%的问答性能
- 适合做跨视频检索与时间定位任务的研究者参考
大型多模态模型在音视频理解方面取得显著进展，但在需要跨大量视频进行复杂推理的实际场景中仍表现不佳。现有视频问答基准普遍仅包含单个视频片段，难以反映真实应用中大规模音视频检索与推理的挑战。为此，我们提出新任务AV-HaystacksQA：根据查询在不同视频中识别出关键片段并串联生成最信息量的回答。为此构建了包含3100个标注问答对的AVHaystacks音视频基准数据集，用于评估大型多模态模型在多视频检索与时间定位任务上的能力。同时提出模型无关的多智能体框架MAGNET，可在我们的数据集上使问答任务的BLEU@4和GPT评分分别相对提升89%和65%。为实现鲁棒评估，引入两个新指标：STEM（捕捉预测步骤序列与真实序列的对齐误差）和MTGS（平衡且可解释的段级定位评估指标）。项目主页：https://schowdhury671.github.io/magnet_project/
原文摘要 · Abstract (English)
Large multimodal models (LMMs) have shown remarkable progress in audio-visual understanding, yet they struggle with real-world scenarios that require complex reasoning across extensive video collections. Existing benchmarks for video question answering remain limited in scope, typically involving one clip per query, which falls short of representing the challenges of large-scale, audio-visual retrieval and reasoning encountered in practical applications. To bridge this gap, we introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer. To this end, we present AVHaystacks, an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task. Additionally, we propose a model-agnostic, multi-agent framework MAGNET to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystacks. To enable robust evaluation of multi-video retrieval and temporal grounding for optimal response generation, we introduce two new metrics, STEM, which captures alignment errors between a ground truth and a predicted step sequence and MTGS, to facilitate balanced and interpretable evaluation of segment-level grounding performance. Project: https://schowdhury671.github.io/magnet_project/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。