将长视频转为图像域，用CLIP增强非连续动作的上下文理解。
Top-down Activity Representation Learning for Video Question Answering
- 把长时间视频转成图像空间，利用CLIP提取非连续动作的视觉上下文。
- 在STAR任务上达78.4%准确率，NExTQA超越当前最优2.8分。
- 适合需要理解复杂事件结构的视频问答场景。
捕捉从原子动作（如拿起礼物、走到沙发、拆礼物）到上下文事件（如庆祝圣诞节）的复杂层级人类活动，对实现高性能视频问答（VideoQA）至关重要。近期工作将多模态模型（如CLIP、LLaVA）扩展至处理连续视频序列，提升了模型的时间推理能力。然而，这些方法往往难以捕捉可分解为多个非连续分布原子动作的上下文事件。为此，本文通过将长期视频序列转换为二维图像域，并微调多模态模型LLaVA以执行VideoQA任务，从而利用CLIP模型的空间视觉上下文表示能力，获取上下文事件的非连续视觉表征。该方法在STAR任务上取得具有竞争力的表现，准确率达78.4%，在NExTQA任务上超越当前最佳结果2.8个百分点。
原文摘要 · Abstract (English)
Capturing complex hierarchical human activities, from atomic actions (e.g., picking up one present, moving to the sofa, unwrapping the present) to contextual events (e.g., celebrating Christmas) is crucial for achieving high-performance video question answering (VideoQA). Recent works have expanded multimodal models (e.g., CLIP, LLaVA) to process continuous video sequences, enhancing the model's temporal reasoning capabilities. However, these approaches often fail to capture contextual events that can be decomposed into multiple atomic actions non-continuously distributed over relatively long-term sequences. In this paper, to leverage the spatial visual context representation capability of the CLIP model for obtaining non-continuous visual representations in terms of contextual events in videos, we convert long-term video sequences into a spatial image domain and finetune the multimodal model LLaVA for the VideoQA task. Our approach achieves competitive performance on the STAR task, in particular, with a 78.4% accuracy score, exceeding the current state-of-the-art score by 2.8 points on the NExTQA task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。