用少量视角图生成任意时间的3D场景,实现昼夜自然过渡。
TimeNeRF: Building Generalizable Neural Radiance Fields across Time from Few-Shot Input Views
- 通过多视角立体视觉与神经辐射场结合,解耦时空特征。
- 仅需少数输入视图即可渲染任意时间的新视角,无需逐场景优化。
- 适合元宇宙、影视特效等需要动态3D环境建模的场景。
我们提出TimeNeRF,一种可泛化的神经渲染方法,能够在少量输入视图下,于任意视角和任意时间合成新视图。现实应用中,采集多视角数据成本高且重新优化效率低。随着元宇宙对沉浸式体验的需求提升,模拟从日出到日落的自然场景变化至关重要。尽管现有基于神经辐射场(NeRF)的技术在新视角合成方面表现优异,但针对时序3D场景建模的研究仍有限,且缺乏专用数据集。为此,我们的方法融合多视角立体视觉、神经辐射场与跨数据集解耦策略,使模型具备少样本泛化能力,构建隐式内容辐射场表示,并支持任意时间点的神经辐射场重建。最终通过体素渲染合成该时间的新视角。实验表明,TimeNeRF可在不进行每场景优化的情况下实现少样本新视角渲染,尤其擅长生成昼夜自然过渡的真实感新视图,精准捕捉从黎明至黄昏的复杂自然变化。
原文摘要 · Abstract (English)
We present TimeNeRF, a generalizable neural rendering approach for rendering novel views at arbitrary viewpoints and at arbitrary times, even with few input views. For real-world applications, it is expensive to collect multiple views and inefficient to re-optimize for unseen scenes. Moreover, as the digital realm, particularly the metaverse, strives for increasingly immersive experiences, the ability to model 3D environments that naturally transition between day and night becomes paramount. While current techniques based on Neural Radiance Fields (NeRF) have shown remarkable proficiency in synthesizing novel views, the exploration of NeRF's potential for temporal 3D scene modeling remains limited, with no dedicated datasets available for this purpose. To this end, our approach harnesses the strengths of multi-view stereo, neural radiance fields, and disentanglement strategies across diverse datasets. This equips our model with the capability for generalizability in a few-shot setting, allows us to construct an implicit content radiance field for scene representation, and further enables the building of neural radiance fields at any arbitrary time. Finally, we synthesize novel views of that time via volume rendering. Experiments show that TimeNeRF can render novel views in a few-shot setting without per-scene optimization. Most notably, it excels in creating realistic novel views that transition smoothly across different times, adeptly capturing intricate natural scene changes from dawn to dusk.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。