提出可控实验框架,系统比较世界模型的记忆机制优劣。
Echo-Memory: A Controlled Study of Memory in Action World Models

- 固定生成器与训练流程,仅变换记忆存储与读取方式。
- 原始上下文记忆在跨场景重访中表现最佳,压缩记忆易丢失关键信息。
- 状态空间循环结构比单纯记忆容量更有效,结构设计至关重要。
我们提出Echo-Memory,一种针对动作条件世界模型中记忆机制的受控研究。这类模型从首帧图像、文本提示和相机动作序列生成多段视频,但其核心缺陷常源于记忆而非局部图像生成:当相机离开并返回后,场景或显著物体可能悄然变化。现有记忆设计难以比较,因性能提升常与主干网络、训练方式、检索策略和评估差异混杂。Echo-Memory固定动作到视频的接口,仅改变生成器对历史信息的存储与读取方式。在共享视频扩散主干、优化器、相机动作表示、采样器和评估流水线的前提下,我们对比了原始上下文、基于压缩的记忆、不同读出路径的空间摘要以及状态空间递归机制。该对照矩阵分离了四个原本混淆的维度:容量、压缩性、读出方式与递归性。我们还通过三分支协议评估记忆能力:重放质量、域内循环重访与开放域返回探测。三类指标常不一致,表明重放保真度不足以代表对世界的记忆。三大发现如下:原始上下文是强容量基线,显著提升开放域返回表现;紧凑性并非容量替代品,激进的空间与混合压缩记忆会丢失返回所需的关键证据;块级状态空间递归在所有机制中对开放域返回最具优势,说明隐式记忆的结构设计与是否使用同等重要。这些结果为超越孤立重放指标的研究提供了简洁协议。
原文摘要 · Abstract (English)
We present \textbf{Echo-Memory}, a controlled study of memory mechanisms in action-conditioned world models. These models generate multi-segment videos from a first frame, text prompt, and camera-action sequence, but their central failure is often memory rather than local image synthesis: after the camera leaves and returns, the scene or salient object may silently change. Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences. Echo-Memory fixes the action-to-video interface and varies only how history is stored and read by the generator. Under a shared video diffusion backbone, optimizer, camera-action representation, sampler, and evaluation pipeline, we compare raw context, compression-based memory, spatial summaries with different read-out paths, and state-space recurrence. This matched matrix separates four otherwise conflated axes: \emph{capacity}, \emph{compression}, \emph{read-out}, and \emph{recurrence}. We also evaluate memory through a three-branch protocol: replay quality, in-domain loop revisit, and open-domain return probes. The branches routinely disagree, showing that replay fidelity is not a sufficient proxy for remembering a world. Three findings follow. Raw context is a strong capacity baseline and improves open-domain return far more than it improves replay metrics. Compactness is not a free substitute for capacity: aggressive spatial and hybrid-compression memories lose the salient evidence needed for return. Finally, block-wise state-space recurrence is the strongest open-domain return mechanism in our matrix, showing that the structure of implicit memory matters as much as the decision to use it. These results provide a compact protocol for studying memory in action world models beyond isolated replay metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。