用相对4D场景图记忆解决长时第一人称视频中的物体追踪难题。
R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video

- 将视频转为按时间、位置和物体关联的可检索记忆条目
- 在EgoLifeQA上问答准确率提升6.7点,时间类问题提升12.5点
- 适合可穿戴助手、AR系统等需要长期物体记忆的场景
长时第一人称视频为可穿戴AI助手提供了丰富数据,但针对物体的追问(如物品被移到哪、何时状态改变、为何移动)仍难解答,因现有基于字幕或转录的记忆难以保持物体身份与结构化空间变化。现有长视频问答方法多关注时间定位与片段检索,而以往3D场景图方法通常依赖强几何信息(如点云、深度图、带位姿的视图),不适用于自由运动的普通RGB视频。R4DSG提出一种相对4D场景图记忆机制,不存储原始图序列,而是将视频转化为以时间、地点、持久物体、锚点相对变化和局部交互上下文为索引的紧凑可查询记忆。核心思路是分离稳定锚点与动态物体,跨帧维持物体身份,并通过锚点相对转移表示物体状态,而非全局对齐的世界模型。该方法基于最新的仅用RGB的提示式视频分割、时序传播与相对3D提升技术,直接生成可用于长时问答的检索型记忆。在EgoLifeQA中255个物体相关问题子集上的评估显示,仅用问题检索时,整体准确率比EgoRAG-Text高6.7点,时间类问题高出12.5点,凸显时序组织物体记忆的价值。结果表明,相对4D场景图是可穿戴助手、AR系统与具身多媒体智能体的实用记忆基础。
原文摘要 · Abstract (English)
Long-horizon egocentric video is a rich substrate for wearable AI assistants, but object-centric questions such as where an item was moved, when it last changed state, or why it was relocated remain difficult because caption- and transcript-based memories rarely preserve persistent object identity or structured spatial change. Existing long-video QA methods mainly emphasize temporal grounding and clip retrieval, while prior 3D scene-graph methods typically assume stronger geometry than free-motion wearable RGB video provides, including point clouds, RGB-D input, posed views, sparse reconstruction, or reconstructed scenes. R4DSG introduces a relative 4D scene graph memory for long egocentric video. Instead of storing raw graph sequences, R4DSG converts video into compact queryable memory entries indexed by time, place, persistent objects, anchor-relative change, and local interaction context. The main idea is to separate stable anchors from dynamic objects, maintain persistent object identity across frames, and represent object state through anchor-relative transitions rather than a globally aligned world model. Built on recent RGB-only advances in promptable video segmentation, temporal propagation, and relative 3D lifting, the method produces a retrieval-ready memory directly usable for long-horizon question answering. Evaluation on a 255-question object-related subset from EgoLifeQA shows, under question-only retrieval, a 6.7-point overall gain over EgoRAG-Text and a 12.5-point gain on when questions, which highlights the value of temporally organized object memory. These results position relative 4D scene graphs as a practical memory substrate for wearable assistants, AR systems, and embodied multimedia agents. GitHub Page: https://dualtransparency.github.io/R4DSG/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。