对比视觉变压器中记忆机制，提升世界模型长程规划能力
On Memory: A comparison of memory mechanisms in world models
- 区分记忆编码与注入两类机制，分析其在残差流中的作用
- 实验显示记忆机制可显著扩展视觉变压器的有效记忆跨度
- 适合研究长时序建模与世界模型规划的开发者参考
世界模型通过基于历史观测和动作预测未来状态，使智能体能在想象环境中进行规划。然而，其长时程规划能力受限于骨干架构的有效记忆跨度，导致长程推演中出现感知漂移，难以实现想象轨迹中的闭环闭合。本文通过分析多种记忆增强机制，研究基于Transformer的世界模型的有效记忆跨度。我们提出一个分类体系，区分记忆编码与记忆注入机制，并从残差流动态角度阐释其扩展记忆的作用。利用状态回忆评估任务，量化各机制的记忆召回能力并分析其权衡。结果表明，记忆机制能有效延长视觉变压器的记忆跨度，为世界模型实现想象轨迹中的闭环闭合提供路径。
原文摘要 · Abstract (English)
World models enable agents to plan within imagined environments by predicting future states conditioned on past observations and actions. However, their ability to plan over long horizons is limited by the effective memory span of the backbone architecture. This limitation leads to perceptual drift in long rollouts, hindering the model's capacity to perform loop closures within imagined trajectories. In this work, we investigate the effective memory span of transformer-based world models through an analysis of several memory augmentation mechanisms. We introduce a taxonomy that distinguishes between memory encoding and memory injection mechanisms, motivating their roles in extending the world model's memory through the lens of residual stream dynamics. Using a state recall evaluation task, we measure the memory recall of each mechanism and analyze its respective trade-offs. Our findings show that memory mechanisms improve the effective memory span in vision transformers and provide a path to completing loop closures within a world model's imagination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。