仅用单目事件流实现非刚性物体的新视角渲染,无需额外RGB数据。
Ev4DGS: Novel-view Rendering of Non-Rigid Objects from Monocular Event Streams
- 通过事件流直接回归可变形3D高斯点云,构建显式图像空间表示
- 在合成与真实数据集上均优于基线方法,重建质量显著提升
- 适合对事件相机、动态物体建模感兴趣的视觉算法研究者
与同步工作的RGB相机相比,事件相机在新视角渲染中具有诸多优势。尽管已有高效事件驱动方法适用于刚性场景,但现有非刚性物体方法仍需稀疏的RGB输入,这带来了实际限制。本文首次探索了仅从事件流学习非刚性物体新视角渲染的可能性,提出Ev4DGS——首个基于单目事件流在显式图像空间(即RGB或灰度图像)中实现非刚性物体新视角渲染的方法。该方法通过两个损失函数:一是将模型输出与二维事件观测空间对齐,二是利用事件生成的二值掩码训练粗粒度3D形变模型。我们在现有合成数据集和新采集的真实数据集上进行了实验对比,结果验证了Ev4DGS的有效性,并在多个基线方法中表现更优。代码与数据集将用于研究目的公开,详见项目页面:https://4dqv.mpi-inf.mpg.de/Ev4DGS/
原文摘要 · Abstract (English)
Event cameras offer various advantages for novel view rendering compared to synchronously operating RGB cameras, and efficient event-based techniques supporting rigid scenes have been recently demonstrated in the literature. In the case of non-rigid objects, however, existing approaches additionally require sparse RGB inputs, which can be a substantial practical limitation; it remains unknown if similar models could be learned from event streams only. This paper sheds light on this challenging open question and introduces Ev4DGS, i.e., the first approach for novel view rendering of non-rigidly deforming objects in the explicit observation space (i.e., as RGB or greyscale images) from monocular event streams. Our method regresses a deformable 3D Gaussian Splatting representation through 1) a loss relating the outputs of the estimated model with the 2D event observation space, and 2) a coarse 3D deformation model trained from binary masks generated from events. We perform experimental comparisons on existing synthetic and newly recorded real datasets with non-rigid objects. The results demonstrate the validity of Ev4DGS and its superior performance compared to multiple naive baselines that can be applied in our setting. We will release our models and the datasets used in the evaluation for research purposes; see the project webpage: https://4dqv.mpi-inf.mpg.de/Ev4DGS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。