通过精准定位关键帧提升视频文字问答准确率
VTAgent: Agentic Keyframe Anchoring for Evidence-Aware Video TextVQA
- 设计代理框架，用问题引导锚定相关关键帧
- 训练自由设置下准确率提升12.12，ANLS提升11.15
- 适合需要高精度视频文本理解的研究者
视频文字视觉问答（Video TextVQA）旨在通过分析视频中的视觉文本内容回答问题。尽管近期视频大模型具备强大多模态理解能力，其在现有基准上的表现仍有限。我们通过逐帧问答的上限分析发现，只要任意一帧正确即可计为正确，该方法显著优于直接视频推理，揭示出性能瓶颈主要在于关键证据的定位，而非推理能力本身。基于此，我们提出一种问题引导的代理框架，先显式锚定相关关键帧再作答。该方法在无训练设置下已超越直接推理，经监督微调（SFT）与强化学习（RL）后，在多个基准上平均准确率提升12.12，ANLS提升11.15，达到新最优结果。研究强调了显式关键帧锚定对推动视频文字问答的重要性。代码将公开。
原文摘要 · Abstract (English)
Video text-based visual question answering (Video TextVQA) aims to answer questions by reasoning over visual textual content appearing in videos. Despite the strong multimodal video understanding capabilities of recent Video-LLMs, their performance on existing Video TextVQA benchmarks remains limited. To better understand this gap, we conduct an upper-bound analysis through frame-wise question answering, counting a sample as correct if any frame yields the right answer, which significantly outperforms direct video-based inference and reveals a substantial performance gap. The results suggest that the primary bottleneck lies in the localization of key question-relevant evidence, rather than in reasoning capacity itself. Building on this insight, we propose a question-guided agent framework that explicitly anchors the relevant keyframes before answering. The approach operates effectively in a training-free setting and consistently surpasses direct video inference. With additional supervised fine-tuning (SFT) and reinforcement learning (RL), it achieves an average improvement of +12.12 in accuracy and +11.15 in ANLS across benchmarks, establishing new state-of-the-art results. Our study underscores the critical role of explicit keyframe anchoring for advancing Video TextVQA. The code will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。