弱监督下实现视频问答的时间定位，无需标注时间信息
TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision
- 通过伪标签与一致性约束，在无时间标注情况下生成答案和时间定位
- 在NExT-GQA、MSVD-QA等基准上达到当前最佳性能
- 适合研究弱监督视频理解与多模态模型的学者参考
我们解决弱监督环境下视频问答中时间定位的问题，无需任何时间标注。给定一段视频和一个问题，模型需生成一个包含起止时间的开放回答。为此，我们提出TOGA：一种用于弱监督下时间定位开放问答的视觉语言模型。通过指令微调，使模型联合生成答案与时间定位结果。在无时间标注的弱监督设置下，我们生成伪时间标签，并通过同一时间段内不同问题的回答一致性来保证标签有效性。实验发现，联合生成答案与定位可同时提升问答与定位性能。我们在基于弱监督的地面实况问答（NExT-GQA）及开放问答（MSVD-QA、ActivityNet-QA）任务上进行评估，均取得当前最优表现。
原文摘要 · Abstract (English)
We address the problem of video question answering (video QA) with temporal grounding in a weakly supervised setup, without any temporal annotations. Given a video and a question, we generate an open-ended answer grounded with the start and end time. For this task, we propose TOGA: a vision-language model for Temporally Grounded Open-Ended Video QA with Weak Supervision. We instruct-tune TOGA to jointly generate the answer and the temporal grounding. We operate in a weakly supervised setup where the temporal grounding annotations are not available. We generate pseudo labels for temporal grounding and ensure the validity of these labels by imposing a consistency constraint between the question of a grounding response and the response generated by a question referring to the same temporal segment. We notice that jointly generating the answers with the grounding improves performance on question answering as well as grounding. We evaluate TOGA on grounded QA and open-ended QA tasks. For grounded QA, we consider the NExT-GQA benchmark which is designed to evaluate weakly supervised grounded question answering. For open-ended QA, we consider the MSVD-QA and ActivityNet-QA benchmarks. We achieve state-of-the-art performance for both tasks on these benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。