动态压缩视频帧,让模型记住更久远的场景细节。
WorldPack: Dynamic Frame Compression for Long-context Video World Modeling
- 根据视角空间相关性动态分配压缩率,避免无关帧占内存。
- 上下文长度从4帧扩展到22帧,推理时间仅增加16%。
- 适合需要长时记忆的导航与空间推理任务,如游戏环境建模。
视频世界模型因其能基于历史观测和导航动作生成高保真未来视觉画面而受到关注。然而,在长时序上保持时空一致性仍是挑战:现有方法要么忽略3D视角几何进行帧压缩,要么仅检索少量空间相关帧而不增加历史总量。本文提出WorldPack,通过引入空间感知的压缩记忆,同时解决上述问题。核心思想是压缩率应非均匀且不固定于时间,而是依据当前视角的3D空间相关性动态分配。该模型包含两个紧密耦合机制:轨迹打包(trajectory packing)通过分层帧压缩将更多历史帧塞入固定上下文;几何选择(geometric selection)利用相机位姿和视场重叠度,为重要空间区域分配更低压缩率。两者结合使有效上下文从4帧扩展至22帧,扩散模型推理时间仅增加16%,视场相关的几何选择带来额外开销。在LoopNav(Minecraft长程一致性基准)及RECON真实导航数据集上评估,结果超越Oasis、Mineworld、DIAMOND和NWM等强基线,尤其在依赖远距离回忆的空间推理任务中表现突出。
原文摘要 · Abstract (English)
Video world models have attracted significant attention for their ability to produce high-fidelity future visual observations conditioned on past observations and navigation actions. However, achieving temporally and spatially consistent generation over long horizons remains an open challenge: existing approaches either compress past frames without explicitly accounting for 3D viewpoint geometry or retrieve only a handful of spatially relevant frames without increasing the total amount of retained history. In this paper, we propose WorldPack, a video world model that introduces spatially-aware compressed memory to address both limitations simultaneously. The key insight is that compression rates should not be uniform or temporally determined, but should instead be dynamically allocated based on 3D spatial relevance to the current viewpoint. WorldPack achieves this through two tightly coupled mechanisms: trajectory packing, which fits substantially more historical frames into a fixed-length context through hierarchical frame compression, and geometric selection, which leverages camera pose information and field-of-view overlap to assign lower compression to spatially important frames and higher compression to less relevant ones. Together, these mechanisms expand the effective context from 4 to 22 frames with moderate computational overhead: trajectory packing increases diffusion-model inference time by 16%, while FoV-based geometric selection introduces an additional cost. We evaluate WorldPack on LoopNav, a Minecraft benchmark for long-horizon spatial consistency, and conduct comprehensive experiments on the RECON, real-world navigation dataset, across multiple metrics. WorldPack outperforms strong baselines, including Oasis, Mineworld, DIAMOND, and NWM, with pronounced gains on spatial reasoning tasks that require recall of distant observations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。