用事件图结构让视频问答更可解释，兼顾准确与透明推理。
ENTER: Event Based Interpretable Reasoning for VideoQA
- 通过事件图构建视频的结构化表示，节点为事件，边为时序/因果关系。
- 在NExT-QA等数据集上优于主流顶向下方法，接近底向上性能。
- 生成可读代码解析推理过程，适合需要可解释性的研究者使用。
本文提出ENTER，一种基于事件图的可解释视频问答系统。事件图将视频转换为图结构，事件作为节点，事件间的时间、因果或层次关系作为边。该结构化表示带来三大优势：1）通过生成解析事件图的代码实现可解释问答；2）在推理过程中融合上下文视觉信息；3）通过层级迭代更新提升鲁棒性。现有可解释方法多为自上而下，忽视底层视觉信息，且脆弱；而自下而上的方法缺乏可解释性。在NExT-QA、IntentQA和EgoSchema数据集上的实验表明，ENTER不仅优于主流顶向下方法，且在性能上媲美底向上方法，更重要的是提供了更优的推理可解释性与可追溯性。
原文摘要 · Abstract (English)
In this paper, we present ENTER, an interpretable Video Question Answering (VideoQA) system based on event graphs. Event graphs convert videos into graphical representations, where video events form the nodes and event-event relationships (temporal/causal/hierarchical) form the edges. This structured representation offers many benefits: 1) Interpretable VideoQA via generated code that parses event-graph; 2) Incorporation of contextual visual information in the reasoning process (code generation) via event graphs; 3) Robust VideoQA via Hierarchical Iterative Update of the event graphs. Existing interpretable VideoQA systems are often top-down, disregarding low-level visual information in the reasoning plan generation, and are brittle. While bottom-up approaches produce responses from visual data, they lack interpretability. Experimental results on NExT-QA, IntentQA, and EgoSchema demonstrate that not only does our method outperform existing top-down approaches while obtaining competitive performance against bottom-up approaches, but more importantly, offers superior interpretability and explainability in the reasoning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。