通过几何引导的多参考纹理迁移,提升通用神经渲染细节表现
GMT: Enhancing Generalizable Neural Rendering via Geometry-Driven Multi-Reference Texture Transfer
- 用射线对齐可变形卷积对齐几何特征,实现跨视角纹理对齐
- 引入纹理保持变压器聚合多视角特征,增强高频细节还原能力
- 即插即用模块,适合需要高保真渲染的通用场景重建任务
新视角合成(NVS)旨在利用多视角图像生成任意视角的图像,近年来基于神经辐射场(NeRF)的研究取得了显著进展。通用NeRF(G-NeRF)解决了传统NeRF每场景优化的难题,通过即时构建辐射场简化了合成流程,更适合实际应用。然而,即使在纹理丰富的多视角输入下,由于缺乏每场景优化,G-NeRF仍难以表现特定场景的精细细节。为此,我们提出一种即插即用的几何驱动多参考纹理迁移网络(GMT)。具体地,提出射线施加的可变形卷积(RayDCN),以反映场景几何的输入与参考特征进行对齐;同时设计纹理保持变压器(TP-Former),在聚合多视角源特征的同时保留纹理信息。该模块使相邻像素在图像增强过程中能直接交互,弥补了原有G-NeRF独立逐像素渲染的缺陷,有效提升高频细节捕捉能力。实验表明,该模块在多个基准数据集上持续提升G-NeRF模型性能。
原文摘要 · Abstract (English)
Novel view synthesis (NVS) aims to generate images at arbitrary viewpoints using multi-view images, and recent insights from neural radiance fields (NeRF) have contributed to remarkable improvements. Recently, studies on generalizable NeRF (G-NeRF) have addressed the challenge of per-scene optimization in NeRFs. The construction of radiance fields on-the-fly in G-NeRF simplifies the NVS process, making it well-suited for real-world applications. Meanwhile, G-NeRF still struggles in representing fine details for a specific scene due to the absence of per-scene optimization, even with texture-rich multi-view source inputs. As a remedy, we propose a Geometry-driven Multi-reference Texture transfer network (GMT) available as a plug-and-play module designed for G-NeRF. Specifically, we propose ray-imposed deformable convolution (RayDCN), which aligns input and reference features reflecting scene geometry. Additionally, the proposed texture preserving transformer (TP-Former) aggregates multi-view source features while preserving texture information. Consequently, our module enables direct interaction between adjacent pixels during the image enhancement process, which is deficient in G-NeRF models with an independent rendering process per pixel. This addresses constraints that hinder the ability to capture high-frequency details. Experiments show that our plug-and-play module consistently improves G-NeRF models on various benchmark datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。