用视觉聚焦机制提升VR中3D高斯点渲染的画质与性能。
VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points
- 外围用低精度高斯点，中心用神经点聚焦细节。
- 在120帧/秒下实现高质量实时渲染，用户偏好度显著提升。
- 适合对画质和性能要求高的虚拟现实应用。
近期基于可微点渲染的新视角合成方法在快速生成逼真场景方面取得了显著进展，如3D高斯点渲染（3DGS）或神经点渲染。然而，这些方法要么需要大量小高斯点，要么需昂贵的逐像素后处理以重建细节，影响渲染效率。为满足虚拟现实（VR）系统的高性能需求，必须控制原始图元或像素数量，从而降低画质。本文提出一种基于视觉聚焦的混合方法，结合两种点渲染的优势，在性能与质量间取得平衡。分析人眼视觉系统后发现，外围区域使用少量、平滑的低精度高斯表示，计算成本低且符合周边视觉感知需求；仅在中央视觉区（视场中心）使用带卷积神经网络的神经点，以小像素开销提供清晰细节。该组合还支持点遮挡剔除等加速技术，降低神经网络负担。评估显示，相比标准的VR就绪3DGS配置，本方法显著提升画面锐度与细节，用户研究中参与者普遍更偏爱该方案。系统满足实时VR交互的性能要求，最终增强用户体验。
原文摘要 · Abstract (English)
Recent advances in novel view synthesis have demonstrated impressive results in fast photorealistic scene rendering through differentiable point rendering, either via Gaussian Splatting (3DGS) [Kerbl and Kopanas et al. 2023] or neural point rendering [Aliev et al. 2020]. Unfortunately, these directions require either a large number of small Gaussians or expensive per-pixel post-processing for reconstructing fine details, which negatively impacts rendering performance. To meet the high performance demands of virtual reality (VR) systems, primitive or pixel counts therefore must be kept low, affecting visual quality. In this paper, we propose a novel hybrid approach based on foveated rendering as a promising solution that combines the strengths of both point rendering directions regarding performance sweet spots. Analyzing the compatibility with the human visual system, we find that using a low-detailed, few primitive smooth Gaussian representation for the periphery is cheap to compute and meets the perceptual demands of peripheral vision. For the fovea only, we use neural points with a convolutional neural network for the small pixel footprint, which provides sharp, detailed output within the rendering budget. This combination also allows for synergistic method accelerations with point occlusion culling and reducing the demands on the neural network. Our evaluation confirms that our approach increases sharpness and details compared to a standard VR-ready 3DGS configuration, and participants of a user study overwhelmingly preferred our method. Our system meets the necessary performance requirements for real-time VR interactions, ultimately enhancing the user's immersive experience. The project page can be found at: https://lfranke.github.io/vr_splatting
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。