根据问题动态选择视频关键帧与变化帧，提升问答准确率
Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answering
- 先探索关键帧与变化帧的分配比例，再按问题选择最优组合
- 在多个基准上最高提升5.8%准确率，且无需模型重训练
- 适合希望提升视频理解效率的研究者与工程师
视频问答得益于视频中的丰富信息，可支持多种应用。然而，长视频生成的大量标记对内存效率和模型性能构成挑战。现有方法通过压缩视频输入缓解此问题，但常忽略不同问题对静态与动态信息的重要性差异，导致有限预算下标记使用效率低下。本文提出一种新型标记选择策略 extsc{explore-then-select}，根据问题需求自适应调整静态与动态信息。框架首先探索关键帧（保留空间细节）与差分帧（捕捉时间变化）之间的不同标记分配，然后利用基于查询感知注意力的度量，无须模型更新即可选出最优标记组合。该方法为即插即用设计，可无缝集成于多种视频语言模型中。大量实验表明，本方法在多个视频问答基准上实现显著性能提升（最高达5.8%）。代码已开源：https://github.com/ANDgate99/Explore-Then-Select。
原文摘要 · Abstract (English)
Video question answering benefits from the rich information in videos, enabling various applications. However, the large volume of tokens generated from long videos presents challenges to memory efficiency and model performance. To alleviate this, existing works propose to compress video inputs, but often overlook the varying importance of static and dynamic information across different queries, leading to inefficient token usage within limited budgets. We propose a novel token selection strategy, \textsc{explore-then-select}, that adaptively adjusts static and dynamic information based on question requirements. Our framework first explores different token allocations between key frames, which preserve spatial details, and delta frames, which capture temporal changes. Then it employs a query-aware attention-based metric to select the optimal token combination without model updates. Our framework is plug-and-play and can be seamlessly integrated within diverse video language models. Extensive experiments show that our method achieves significant performance improvements (up to 5.8\%) on multiple video question answering benchmarks. Our code is available at https://github.com/ANDgate99/Explore-Then-Select .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。