让3D场景随时间动起来,还能用语言精准查询物体在何时何地如何运动。
4D Synchronized Fields: Motion-Language Gaussian Splatting for Temporal Scene Understanding
- 用可解释的物体级运动分解构建4D高斯表示,边重建边学运动结构。
- 在HyperNeRF上达到28.52 dB PSNR,时序语义检索准确率超现有方法近30%。
- 首次实现单一模型中同时输出可理解的运动原型与时间对齐的语言场,适合交互式时空查询。
现有4D表示将几何、运动与语义解耦:重建方法忽略可解释的运动结构;语言引导方法在运动学习后附加语义,无法感知物体如何移动;运动感知方法将动态编码为不透明的逐点残差,缺乏物体级组织。本文提出4D Synchronized Fields,一种4D高斯表示,能在重建过程中内联学习物体因子化的运动,并通过每物体条件化的场同步语言到生成的运动学特征。每个高斯轨迹分解为共享物体运动与隐式残差,运动学条件化的脊图预测时间语义变化,形成统一表示,使重建、运动与语义在结构上耦合,支持开放词汇的时序查询以检索物体与时刻。在HyperNeRF上,4D Synchronized Fields取得28.52 dB平均PSNR,高于所有语言引导与运动感知基线,仅比纯重建方法低1.5 dB。在定向时序状态检索任务中,该模型达到0.884平均准确率、0.815平均vIoU、0.733平均tIoU,显著优于4D LangSplat(0.620, 0.433, 0.439)和LangSplat(0.415, 0.304, 0.262)。消融实验表明,运动学条件化是主要驱动力,相比静态嵌入基线提升0.45 tIoU。4D Synchronized Fields是唯一能从单个训练模型中同时揭示可解释运动原语与时间对齐语言场的方法。代码将开源。
原文摘要 · Abstract (English)
Current 4D representations decouple geometry, motion, and semantics: reconstruction methods discard interpretable motion structure; language-grounded methods attach semantics after motion is learned, blind to how objects move; and motion-aware methods encode dynamics as opaque per-point residuals without object-level organization. We propose 4D Synchronized Fields, a 4D Gaussian representation that learns object-factored motion in-loop during reconstruction and synchronizes language to the resulting kinematics through a per-object conditioned field. Each Gaussian trajectory is decomposed into shared object motion plus an implicit residual, and a kinematic-conditioned ridge map predicts temporal semantic variation, yielding a single representation in which reconstruction, motion, and semantics are structurally coupled and enabling open-vocabulary temporal queries that retrieve both objects and moments. On HyperNeRF, 4D Synchronized Fields achieves 28.52 dB mean PSNR, the highest among all language-grounded and motion-aware baselines, within 1.5 dB of reconstruction-only methods. On targeted temporal-state retrieval, the kinematic-conditioned field attains 0.884 mean accuracy, 0.815 mean vIoU, and 0.733 mean tIoU, surpassing 4D LangSplat (0.620, 0.433, and 0.439 respectively) and LangSplat (0.415, 0.304, and 0.262). Ablation confirms that kinematic conditioning is the primary driver, accounting for +0.45 tIoU over a static-embedding-only baseline. 4D Synchronized Fields is the only method that jointly exposes interpretable motion primitives and temporally grounded language fields from a single trained representation. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。