让智能体用自然语言查询长时间跨度的物体运动轨迹。
Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents
- 用语言描述+稀疏空间锚点+视觉锚点融合压缩物体运动历史。
- 在24小时视频上实现8.7到26.1倍压缩，查询延迟低于1秒。
- 支持跨天物体与语义轨迹检索，适合长期任务智能体。
执行长周期任务的具身智能体需要一种记忆表征，使动态物体的状态变化在数小时至数天的观察期内仍可通过自然语言进行查询。现有系统要么丢失精细运动信息（片段级视频-语言嵌入），仅保留原始坐标（几何SLAM），或仅围绕当前任务上下文组织记忆（智能体工作记忆）。这些方法均无法为每个物体提供可语言查询的状态时间线。本文提出关键创新：语言轨迹编码（Linguistic Trajectory Encoding, LTE），通过融合自然语言描述、稀疏空间锚点和视觉锚点，压缩动态物体的运动历史。LTE根据运动复杂度自适应压缩：在无可靠观测时段锚定于最后可见位置，运动阶段则用几何路标点与语言描述结合以保持精度。为评估其跨长时间的能力，我们构建了空间记忆基准（Spatial Memory Benchmark, SMB），基于EgoLife多日录制数据，聚焦现有基准缺失的能力：语义轨迹检索与长时程物体检索。在SMB上，基于LTE的系统在语义轨迹检索中达45.3%成功率，在长时程物体检索中达48.7%，优于结构化记忆与视觉语言模型基线（最佳前序：31.9%与34.4%）。在24小时视频上，LTE实现8.7×至26.1×的轨迹压缩，查询延迟低于1秒。在Ego4D自然语言查询任务中，系统达到R@1 28.75% / R@5 55.10%，较EgoVLPv2提升15.80 / 31.30个百分点。
原文摘要 · Abstract (English)
Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\%$ success in semantic trajectory retrieval and $48.7\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\%$ and $34.4\%$). LTE achieves trajectory compression by factors of $8.7\times$ to $26.1\times$ with sub-second query latency on $24$\,h video. On Ego4D natural-language queries, the system reaches $28.75\%$ / $55.10\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。