让模型同时回答问题并定位文本区域，提升视频问答可解释性。
Scene-Text Grounding for Text-Based Video Question Answering
- 通过时空对比学习实现弱监督下的文本定位与问答
- 在新数据集上现有方法表现差，人类仍有巨大提升空间
- 适合关注可解释AI、视频理解与文本定位的研究者
现有文本驱动的视频问答（TextVideoQA）因决策过程不透明且过度依赖场景文本识别而受到批评。本文提出结构化文本视频问答（Grounded TextVideoQA），要求模型在回答问题的同时，准确定位相关场景文本区域，从而将问答与场景文本识别解耦，推动可解释性研究。该任务具有三重意义：一是促使模型依赖场景文本证据而非捷径；二是直接以文本区域作为视觉答案，避免严格字符串匹配带来的评估失效；三是分离视频问答与场景文本识别的挑战，便于诊断失败原因。为此，我们提出T2S-QA模型，采用解耦的时序-空间对比学习策略，在弱监督下实现场景文本定位与问答。为支持评估，我们构建新数据集ViTXT-GQA，包含52,000个场景文本边界框，覆盖2,200个时间片段、2,000个问题和729个视频。实验表明现有技术在该任务上存在严重局限。尽管T2S-QA表现更优，但与人类水平仍有显著差距。对理想文本输入的分析表明，主要瓶颈在于场景文本识别本身。
原文摘要 · Abstract (English)
Existing efforts in text-based video question answering (TextVideoQA) are criticized for their opaque decisionmaking and heavy reliance on scene-text recognition. In this paper, we propose to study Grounded TextVideoQA by forcing models to answer questions and spatio-temporally localize the relevant scene-text regions, thus decoupling QA from scenetext recognition and promoting research towards interpretable QA. The task has three-fold significance. First, it encourages scene-text evidence versus other short-cuts for answer predictions. Second, it directly accepts scene-text regions as visual answers, thus circumventing the problem of ineffective answer evaluation by stringent string matching. Third, it isolates the challenges inherited in VideoQA and scene-text recognition. This enables the diagnosis of the root causes for failure predictions, e.g., wrong QA or wrong scene-text recognition? To achieve Grounded TextVideoQA, we propose the T2S-QA model that highlights a disentangled temporal-to-spatial contrastive learning strategy for weakly-supervised scene-text grounding and grounded TextVideoQA. To facilitate evaluation, we construct a new dataset ViTXT-GQA which features 52K scene-text bounding boxes within 2.2K temporal segments related to 2K questions and 729 videos. With ViTXT-GQA, we perform extensive experiments and demonstrate the severe limitations of existing techniques in Grounded TextVideoQA. While T2S-QA achieves superior results, the large performance gap with human leaves ample space for improvement. Our further analysis of oracle scene-text inputs posits that the major challenge is scene-text recognition. To advance the research of Grounded TextVideoQA, our dataset and code are at https://github.com/zhousheng97/ViTXT-GQA.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。