arXiv:2604.04372cs.CV2026-04中稿 · CVPR被引 1

将外部知识以视觉化图谱形式融合,让视频推理更清晰可查。

Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoning

  • 用视觉空间图谱替代文本检索,统一知识与视频的表示
  • 在多个基准上提升推理性能,知识密集型任务提升更明显
  • 无需训练、可追溯证据链,适合需要可解释性的应用

当视频推理需外部知识时,现有大模型常通过文本或多片段检索补充上下文,但异质信号混入单一注意力空间会导致注意力分散和认知负荷增加,即使在非长视频中亦然。问题不仅在于检索什么,更在于如何表示和融合知识。本文提出Graph-to-Frame RAG(G2F-RAG),一种无需训练且可审计的范式:离线阶段,智能体构建不依赖具体问题的视频知识图谱,整合实体、事件、空间关系及关联世界知识;在线阶段,分层多智能体控制器判断是否需要外部知识,检索最小必要子图,并将其渲染为单个推理帧附加至视频。大模型在统一视觉域中进行联合推理,降低认知负担并保留明确可查的证据路径。G2F-RAG兼容多种骨干模型且可扩展,在多个公开基准上实现一致提升,尤其在知识密集型任务中表现更优。消融实验进一步验证知识表示与传递方式的重要性。该方法将检索重构为视觉空间的知识融合,提升视频推理的鲁棒性与可解释性。

原文摘要 · Abstract (English)

When video reasoning requires external knowledge, many systems with large multimodal models (LMMs) adopt retrieval augmentation to supply the missing context. Appending textual or multi-clip evidence, however, forces heterogeneous signals into a single attention space. We observe diluted attention and higher cognitive load even on non-long videos. The bottleneck is not only what to retrieve but how to represent and fuse external knowledge with the video backbone.We present Graph-to-Frame RAG (G2F-RAG), a training free and auditable paradigm that delivers knowledge in the visual space. On the offline stage, an agent builds a problem-agnostic video knowledge graph that integrates entities, events, spatial relations, and linked world knowledge. On the online stage, a hierarchical multi-agent controller decides whether external knowledge is needed, retrieves a minimal sufficient subgraph, and renders it as a single reasoning frame appended to the video. LMMs then perform joint reasoning in a unified visual domain. This design reduces cognitive load and leaves an explicit, inspectable evidence trail.G2F-RAG is plug-and-play across backbones and scales. It yields consistent gains on diverse public benchmarks, with larger improvements in knowledge-intensive settings. Ablations further confirm that knowledge representation and delivery matter. G2F-RAG reframes retrieval as visual space knowledge fusion for robust and interpretable video reasoning.

视频推理知识融合可解释性图神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。