让手机实时渲染高质量复杂3D场景,兼顾速度与内存。
NeRFlex: Resource-aware Real-time High-quality Rendering of Complex Scenes on Mobile Devices
- 将场景拆成多个子场景,每个用独立NeRF表示
- 动态规划配置参数,在手机内存和算力限制下实现实时渲染
- 适合移动端3D重建、AR应用开发人员参考
神经辐射场(NeRF)是用于3D重建中新视角合成的前沿神经网络技术,但其高计算需求制约了在移动设备上的部署。尽管基于网格的NeRF方案已在移动端实现实时渲染,但在处理实际复杂场景时往往难以保证高质量重建,且预计算中间结果带来的显著内存开销也影响实用性。为此,我们提出NeRFlex,一个面向移动设备的资源感知、高分辨率、实时渲染框架。NeRFlex结合移动端NeRF渲染与多NeRF表示,将场景分解为多个子场景,每个由独立的NeRF网络建模。关键在于,它将内存与计算约束作为首要考虑因素,并重构重建流程。首先设计细节导向的分割模块,识别高频细节子场景;针对每个NeRF网络,构建基于领域知识的轻量级性能评估器,精准映射配置到视觉质量与内存占用。基于此,结合移动设备资源约束,提出一种动态规划算法,高效求解所有NeRF表示的最优配置,克服原问题的NP-hard特性。在真实世界数据集和商用移动设备上的大量实验表明,NeRFlex实现了商业手机上的实时、高质量渲染。
原文摘要 · Abstract (English)
Neural Radiance Fields (NeRF) is a cutting-edge neural network-based technique for novel view synthesis in 3D reconstruction. However, its significant computational demands pose challenges for deployment on mobile devices. While mesh-based NeRF solutions have shown potential in achieving real-time rendering on mobile platforms, they often fail to deliver high-quality reconstructions when rendering practical complex scenes. Additionally, the non-negligible memory overhead caused by pre-computed intermediate results complicates their practical application. To overcome these challenges, we present NeRFlex, a resource-aware, high-resolution, real-time rendering framework for complex scenes on mobile devices. NeRFlex integrates mobile NeRF rendering with multi-NeRF representations that decompose a scene into multiple sub-scenes, each represented by an individual NeRF network. Crucially, NeRFlex considers both memory and computation constraints as first-class citizens and redesigns the reconstruction process accordingly. NeRFlex first designs a detail-oriented segmentation module to identify sub-scenes with high-frequency details. For each NeRF network, a lightweight profiler, built on domain knowledge, is used to accurately map configurations to visual quality and memory usage. Based on these insights and the resource constraints on mobile devices, NeRFlex presents a dynamic programming algorithm to efficiently determine configurations for all NeRF representations, despite the NP-hardness of the original decision problem. Extensive experiments on real-world datasets and mobile devices demonstrate that NeRFlex achieves real-time, high-quality rendering on commercial mobile devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。