用图结构建模视频中人与物的互动，提升问答准确率。
GHR-VQA: Graph-guided Hierarchical Relational Reasoning for Video Question Answering
- 以场景图为框架，将每帧转为图结构，通过人类节点串联跨帧关系。
- 在AGQA数据集上，对象关系推理性能比现有方法高7.3%。
- 适合关注视频理解中人类行为分析的研究者和开发者。
我们提出GHR-VQA，一种基于图的层次化关系推理框架，用于视频问答（Video QA）。不同于传统像素级方法，该框架将每一帧表示为场景图，并将跨帧的人类节点连接至全局根节点，形成视频级图结构，实现以人类为中心的跨帧推理。视频级图由图神经网络（GNN）处理，生成富含上下文信息的嵌入表示，再与问题特征在分层网络中融合，提升对视频内容的局部与全局理解。该显式的人类-根结构增强了可解释性，将动作分解为人类-物体交互，深化了对时空动态的理解。我们在Action Genome Question Answering（AGQA）数据集上验证了该方法，显著提升性能，对象关系推理方面相比现有最佳方法提升7.3%。
原文摘要 · Abstract (English)
We propose GHR-VQA, Graph-guided Hierarchical Relational Reasoning for Video Question Answering (Video QA), a novel human-centric framework that incorporates scene graphs to capture intricate human-object interactions within video sequences. Unlike traditional pixel-based methods, each frame is represented as a scene graph and human nodes across frames are linked to a global root, forming the video-level graph and enabling cross-frame reasoning centered on human actors. The video-level graphs are then processed by Graph Neural Networks (GNNs), transforming them into rich, context-aware embeddings for efficient processing. Finally, these embeddings are integrated with question features in a hierarchical network operating across different abstraction levels, enhancing both local and global understanding of video content. This explicit human-rooted structure enhances interpretability by decomposing actions into human-object interactions and enables a more profound understanding of spatiotemporal dynamics. We validate our approach on the Action Genome Question Answering (AGQA) dataset, achieving significant performance improvements, including a 7.3% improvement in object-relation reasoning over the state of the art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。