让神经渲染在边缘设备上高效运行,内存压缩21倍且画质不降。
SpNeRF: Memory Efficient Sparse Volumetric Neural Rendering Accelerator for Edge Devices
- 用哈希映射和位图掩码实现稀疏体素网格的低内存存储与高效解码。
- 内存减少21.07倍,峰值信噪比损失小于0.5dB,支持实时渲染。
- 专为边缘设备设计软硬件协同架构,能效提升超600倍,适合AR/VR部署。
神经渲染因高质量输出在增强现实与虚拟现实应用中备受关注,但其庞大的体素网格数据与不规则访问模式给边缘设备实时处理带来挑战。现有工作虽关注数据局部性,却未解决体素网格过大导致的频繁片外访问与高片上内存需求问题。本文提出SpNeRF,一种面向稀疏体素神经渲染的软硬件协同解决方案。通过分析体素网格内在稀疏性,我们设计新型预处理与在线解码流程,显著降低体素网格内存占用。预处理阶段采用哈希映射支持不规则访问,同时保持最小内存开销;在线解码引入位图掩码,缓解哈希冲突带来的峰值信噪比损失。进一步地,设计专用硬件架构以支持该稀疏体素处理技术。实验表明,SpNeRF平均内存减少21.07倍,画质保持相近(PSNR损失<0.5dB)。相较于Jetson XNX、Jetson ONX、RT-NeRF.Edge和NeuRex.Edge,分别实现95.1×、63.5×、1.5×和10.3×的加速,能效提升625.6×、529.1×、4×和4.4×。
原文摘要 · Abstract (English)
Neural rendering has gained prominence for its high-quality output, which is crucial for AR/VR applications. However, its large voxel grid data size and irregular access patterns challenge real-time processing on edge devices. While previous works have focused on improving data locality, they have not adequately addressed the issue of large voxel grid sizes, which necessitate frequent off-chip memory access and substantial on-chip memory. This paper introduces SpNeRF, a software-hardware co-design solution tailored for sparse volumetric neural rendering. We first identify memory-bound rendering inefficiencies and analyze the inherent sparsity in the voxel grid data of neural rendering. To enhance efficiency, we propose novel preprocessing and online decoding steps, reducing the memory size for voxel grid. The preprocessing step employs hash mapping to support irregular data access while maintaining a minimal memory size. The online decoding step enables efficient on-chip sparse voxel grid processing, incorporating bitmap masking to mitigate PSNR loss caused by hash collisions. To further optimize performance, we design a dedicated hardware architecture supporting our sparse voxel grid processing technique. Experimental results demonstrate that SpNeRF achieves an average 21.07$\times$ reduction in memory size while maintaining comparable PSNR levels. When benchmarked against Jetson XNX, Jetson ONX, RT-NeRF.Edge and NeuRex.Edge, our design achieves speedups of 95.1$\times$, 63.5$\times$, 1.5$\times$ and 10.3$\times$, and improves energy efficiency by 625.6$\times$, 529.1$\times$, 4$\times$, and 4.4$\times$, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。