ReWorld实现长时记忆下的实时交互世界模型,支持分钟级回溯与稳定视频生成。
ReWorld: An Interactive World Model with Long-Horizon Memory

- 用分头注意力机制分离短期控制与长期记忆,全局头+随机路由避免能力固化。
- 在64秒长轨迹中仍能准确复现起始视角,12块缓存下超越滑动窗口与全量缓存。
- 适用于需要高保真实时交互的虚拟环境构建,如游戏、元宇宙场景开发。
交互式世界模型需响应用户操作、记住已访问位置并实时流式输出。该矛盾源于控制要求短时程,记忆则需无界历史。ReWorld在训练中分离两者,在推理时施加固定预算:使用基于位姿索引的地标库配合有限缓存,仅检索最近地标的视觉信息。一个对齐物理尺度的数据引擎将八类数据(虚幻渲染飞行、游戏漫游、真实影像等)统一到同一动作尺度,确保相同按键移动相同距离;回文轨迹提供记忆训练所需的回访证据。通过仅适配LoRA的分布匹配蒸馏,采样步数压缩至四步,单一主干模型兼顾高保真多步生成与实时交互,可流畅生成704×1280分辨率的写实、游戏风与风格化世界视频。在覆盖动作跟随、长时回忆与画质的三轴评估中,相较六种前沿模型,其旋转误差低至11.95°,相机运动一致性最佳,且在64秒、384个潜在变量的往返滚动生成中,12块缓存仍能完整复现起始视图——而滑动窗口早已淘汰证据,全键值注意力已超内存限制。
原文摘要 · Abstract (English)
An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity ($11.95^\circ$ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts ($64$\,s, $384$ latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。