arXiv:2410.06756cs.CV2024-10NeurIPS被引 70

用稀疏控制点驱动网格与高斯混合,实现单视频生成高质量4D动态物体。

DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation

  • 结合网格与高斯点,用几何绑定方式优化表面和形状。
  • 通过双四元数与线性混合皮肤算法,提升运动一致性与形变质量。
  • 兼容现代图形管线,适合游戏与影视领域的4D内容生成。

近期2D/3D生成技术推动了从单目视频生成动态3D物体的发展。以往方法多依赖隐式神经辐射场(NeRF)或显式高斯点云作为基础表示,难以兼顾时空一致性和表面外观。受现代3D动画流程启发,本文提出DreamMesh4D,一种融合网格表示与几何绑定技术的新框架,可从单目视频生成高质量4D物体。不使用传统贴图,而是将高斯点绑定到网格三角面片上,实现纹理与顶点的可微优化。该方法首先通过图像到3D生成获取粗略网格,再在网格表面均匀采样稀疏控制点,构建变形图以提升效率并提供额外约束。每一步中,变形网络预测稀疏点变换,通过新型混合几何皮肤算法(融合LBS与DQS)同步更新网格顶点与表面高斯点,有效缓解单一方法缺陷。静态表面高斯点、网格顶点及变形网络通过参考视图光度损失、得分蒸馏损失及其他正则项,在两阶段中联合训练。大量实验表明该方法性能优越,且兼容现代图形工作流,具备在3D游戏与电影工业中的应用潜力。

原文摘要 · Abstract (English)

Recent advancements in 2D/3D generative techniques have facilitated the generation of dynamic 3D objects from monocular videos. Previous methods mainly rely on the implicit neural radiance fields (NeRF) or explicit Gaussian Splatting as the underlying representation, and struggle to achieve satisfactory spatial-temporal consistency and surface appearance. Drawing inspiration from modern 3D animation pipelines, we introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Instead of utilizing classical texture map for appearance, we bind Gaussian splats to triangle face of mesh for differentiable optimization of both the texture and mesh vertices. In particular, DreamMesh4D begins with a coarse mesh obtained through an image-to-3D generation procedure. Sparse points are then uniformly sampled across the mesh surface, and are used to build a deformation graph to drive the motion of the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using a deformation network, and the mesh vertices as well as the surface Gaussians are deformed via a novel geometric skinning algorithm, which is a hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well as the deformation network are learned via reference view photometric loss, score distillation loss as well as other regularizers in a two-stage manner. Extensive experiments demonstrate superior performance of our method. Furthermore, our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.

4D生成网格生成高斯渲染视频生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。