用深度3D重建解决大视差图像拼接难题,保持几何完整性。
PIS3R: Very Large Parallax Image Stitching via Deep 3D Reconstruction
- 基于视觉几何的Transformer实现深度3D重建与相机参数估计。
- 重建点云重投影生成初始拼接图,支持大视差场景。
- 引入点条件扩散模块修复孔洞噪声,适合3D视觉任务下游应用。
图像拼接旨在将不同视角拍摄的两张图像对齐为一张无缝、更宽的图像。然而当场景存在深度变化且相机基线较大时,会出现显著视差——即场景元素在不同视角下的相对位置差异明显。现有拼接方法难以有效处理此类大视差图像。本文提出PIS3R,一种基于深度3D重建的拼接方法,可稳健应对极大视差。首先,利用基于视觉几何的Transformer对具有大视差的输入图像进行处理,获得相机内参、外参及稠密3D场景重建。随后,利用恢复的相机参数将重建的稠密点云重投影至指定参考视图,实现像素级对齐并生成初始拼接图。最后,为进一步消除初始拼接中的空洞或噪声,提出点条件图像扩散模块以获得精细化结果。实验表明,该方法在大视差图像拼接上表现优异,定性与定量均优于现有方法,且完全保留了所有像素在3D摄影测量上下文中的几何一致性,可直接用于下游3D视觉任务(如SfM)。
原文摘要 · Abstract (English)
Image stitching aim to align two images taken from different viewpoints into one seamless, wider image. However, when the 3D scene contains depth variations and the camera baseline is significant, noticeable parallax occurs-meaning the relative positions of scene elements differ substantially between views. Most existing stitching methods struggle to handle such images with large parallax effectively. To address this challenge, in this paper, we propose an image stitching solution called PIS3R that is robust to very large parallax based on the novel concept of deep 3D reconstruction. First, we apply visual geometry grounded transformer to two input images with very large parallax to obtain both intrinsic and extrinsic parameters, as well as the dense 3D scene reconstruction. Subsequently, we reproject reconstructed dense point cloud onto a designated reference view using the recovered camera parameters, achieving pixel-wise alignment and generating an initial stitched image. Finally, to further address potential artifacts such as holes or noise in the initial stitching, we propose a point-conditioned image diffusion module to obtain the refined result.Compared with existing methods, our solution is very large parallax tolerant and also provides results that fully preserve the geometric integrity of all pixels in the 3D photogrammetric context, enabling direct applicability to downstream 3D vision tasks such as SfM. Experimental results demonstrate that the proposed algorithm provides accurate stitching results for images with very large parallax, and outperforms the existing methods qualitatively and quantitatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。