构建用户中心的持续空间智能推理框架，提升第一人称视频中物体定位与记忆能力。
Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams
- 基于流式第一人称视觉数据，增量构建结构化空间记忆。
- 在8.1K+时间戳问题上，显著提升主流多模态大模型的空间推理准确率。
- 适合研究第一人称智能助手、长期视觉记忆与动态场景理解的学者。
我们提出UCS-Bench，一个涵盖170+小时第一人称视觉观测、包含8.1K+时间戳问题的数据集，用于诊断第一人称视频流中的用户中心持续空间智能。该任务强调动态空间推理、长期记忆及其与用户实时位置的对齐。我们提出DirectMe框架，从持续的视角观测中增量构建并维护结构化空间记忆。DirectMe能以用户移动为基准，实现物体位置的鲁棒追踪与回忆。通过将视觉感知、记忆更新与空间推理紧密耦合，该方法支持长时程查询，包括交互回忆、视角歧义消解及动态场景适应。实验表明，DirectMe显著提升了主流多模态大模型的空间推理能力，超越多种空间感知与长序列视频模型。我们希望该基准与解决方案能推动第一人称智能助手的空间智能研究。数据与代码见https://github.com/cocowy1/UCS-Bench。
原文摘要 · Abstract (English)
We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users' real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user's movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。