直接支持鱼眼输入,避免畸变校正导致的模糊与浮点伪影。
DirectFisheye-GS: Enabling Native Fisheye Input in Gaussian Splatting with Cross-View Joint Optimization

- 在3D高斯泼溅中原生接入鱼眼图像,跳过畸变校正预处理。
- 通过跨视角联合优化,消除边缘区域的浮点和拉伸伪影。
- 适用于鱼眼与针孔相机通用,提升低频区域重建质量。
3D高斯泼溅(3DGS)已实现从日常图像中高效重建3D场景,支持实时高保真渲染,显著推动了VR/AR应用发展。鱼眼相机凭借更宽的视场(FOV),有望以更少输入实现高质量重建,近来备受关注。然而,由于3DGS依赖光栅化,现有方法通常先对鱼眼图像进行去畸变再训练,带来两个问题:1)图像边缘出现黑边,造成信息丢失,抵消鱼眼大视场优势;2)去畸变过程中的拉伸与重采样使每个像素值扩散至更大区域,稀释细节密度,导致3DGS在低频区过拟合,产生模糊与浮点伪影。本文将鱼眼相机模型直接融入原始3DGS框架,实现无需预处理的原生鱼眼图像输入。尽管建模正确,仍发现边缘区域存在浮点现象:畸变随边缘加剧,且3DGS原有每轮随机选图优化忽略了高斯点的跨视角关联性,导致其形状极端(如过大或拉长),降低重建质量。为此,我们提出一种基于特征重叠的跨视角联合优化策略,在多视角间建立一致的几何与光照约束,该方法亦可应用于现有针孔相机流程。DirectFisheye-GS在公开数据集上表现达到或超越当前最优水平。
原文摘要 · Abstract (English)
3D Gaussian Splatting (3DGS) has enabled efficient 3D scene reconstruction from everyday images with real-time, high-fidelity rendering, greatly advancing VR/AR applications. Fisheye cameras, with their wider field of view (FOV), promise high-quality reconstructions from fewer inputs and have recently attracted much attention. However, since 3DGS relies on rasterization, most subsequent works involving fisheye camera inputs first undistort images before training, which introduces two problems: 1) Black borders at image edges cause information loss and negate the fisheye's large FOV advantage; 2) Undistortion's stretch-and-interpolate resampling spreads each pixel's value over a larger area, diluting detail density -- causes 3DGS overfitting these low-frequency zones, producing blur and floating artifacts. In this work, we integrate fisheye camera model into the original 3DGS framework, enabling native fisheye image input for training without preprocessing. Despite correct modeling, we observed that the reconstructed scenes still exhibit floaters at image edges: Distortion increases toward the periphery, and 3DGS's original per-iteration random-selecting-view optimization ignores the cross-view correlations of a Gaussian, leading to extreme shapes (e.g., oversized or elongated) that degrade reconstruction quality. To address this, we introduce a feature-overlap-driven cross-view joint optimization strategy that establishes consistent geometric and photometric constraints across views-a technique equally applicable to existing pinhole-camera-based pipelines. Our DirectFisheye-GS matches or surpasses state-of-the-art performance on public datasets. Project Page: https://yzxqh.github.io/DirectFisheye-GS/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。