分离静态与动态记忆,实现动态场景高保真重建
Mem4D: Decoupling Static and Dynamic Memory for Dynamic Scene Reconstruction
- 用双记忆架构分别处理静态结构和动态运动
- 在多个基准上达到顶尖性能,静态无漂移、动态更清晰
- 适合需要精确动态场景重建的研究者和开发者
从单目视频重建动态场景的稠密几何结构是一项关键但极具挑战的任务。现有基于记忆的方法虽能实现高效在线重建,却面临内存需求困境:长期稳定性要求与动态细节高保真保留之间存在根本矛盾。这一矛盾迫使现有方法妥协,导致静态结构出现几何漂移或动态物体重建模糊失真。为解决此问题,我们提出 Mem4D,一种解耦静态几何与动态运动建模的新框架。受此启发,设计双记忆架构:1)瞬态动态记忆(TDM)聚焦于近期帧的高频运动细节,实现动态内容的精确细粒度建模;2)持久结构记忆(PSM)压缩并保留长期空间信息,确保静态元素的全局一致性与无漂移重建。通过交替查询这两个专用记忆模块,Mem4D 同时保持静态几何的全局一致性和动态元素的高保真重建。在多个挑战性基准上的实验表明,该方法达到当前最优或具有竞争力的性能,同时保持高效率。代码将公开。
原文摘要 · Abstract (English)
Reconstructing dense geometry for dynamic scenes from a monocular video is a critical yet challenging task. Recent memory-based methods enable efficient online reconstruction, but they fundamentally suffer from a Memory Demand Dilemma: The memory representation faces an inherent conflict between the long-term stability required for static structures and the rapid, high-fidelity detail retention needed for dynamic motion. This conflict forces existing methods into a compromise, leading to either geometric drift in static structures or blurred, inaccurate reconstructions of dynamic objects. To address this dilemma, we propose Mem4D, a novel framework that decouples the modeling of static geometry and dynamic motion. Guided by this insight, we design a dual-memory architecture: 1) The Transient Dynamics Memory (TDM) focuses on capturing high-frequency motion details from recent frames, enabling accurate and fine-grained modeling of dynamic content; 2) The Persistent Structure Memory (PSM) compresses and preserves long-term spatial information, ensuring global consistency and drift-free reconstruction for static elements. By alternating queries to these specialized memories, Mem4D simultaneously maintains static geometry with global consistency and reconstructs dynamic elements with high fidelity. Experiments on challenging benchmarks demonstrate that our method achieves state-of-the-art or competitive performance while maintaining high efficiency. Codes will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。