解决视频拼接中的动态抖动问题,实现稳定与精准同步优化。
StabStitch++: Unsupervised Online Video Stitching with Spatiotemporal Bidirectional Warps
- 引入虚拟中平面和双向可微分解,均衡两视图对齐负担。
- 设计时空联合平滑模型,显著降低拼接视频的动态抖动。
- 支持实时在线拼接,适合无人机、VR等动态拍摄场景。
我们将视频拼接问题聚焦于新出现的‘形变抖动’现象,即在将图像拼接扩展至视频拼接时,因连续不平滑的形变导致的时间内容抖动。即使输入视频稳定,拼接结果仍不可避免产生不良形变抖动,影响视觉体验。为此,我们提出StabStitch++,一个基于无监督学习的新型视频拼接框架,可同时实现空间拼接与时间稳定。不同于传统方法仅将一图像映射到另一图像,我们假设原图像平面之间存在虚拟中平面,并将其投影至该平面;设计可微双向分解模块,解耦单应变换并融入空间形变,均匀分担对齐压力与投影畸变。受视频稳定中相机路径启发,我们推导出拼接轨迹的数学表达式,融合时空形变。最后,提出形变平滑模型,采用混合损失函数,同时促进内容对齐、轨迹平滑与在线协作。相比牺牲对齐换取稳定的StabStitch,StabStitch++无需权衡,双目标同时优化,尤其在线模式下表现更优。为建立评估基准并训练模型,我们构建了一个包含丰富相机运动与场景变化的视频拼接数据集。实验表明,StabStitch++在拼接性能、鲁棒性与效率上均超越现有方法,推动该领域向实时在线系统迈进。
原文摘要 · Abstract (English)
We retarget video stitching to an emerging issue, named warping shake, which unveils the temporal content shakes induced by sequentially unsmooth warps when extending image stitching to video stitching. Even if the input videos are stable, the stitched video can inevitably cause undesired warping shakes and affect the visual experience. To address this issue, we propose StabStitch++, a novel video stitching framework to realize spatial stitching and temporal stabilization with unsupervised learning simultaneously. First, different from existing learning-based image stitching solutions that typically warp one image to align with another, we suppose a virtual midplane between original image planes and project them onto it. Concretely, we design a differentiable bidirectional decomposition module to disentangle the homography transformation and incorporate it into our spatial warp, evenly spreading alignment burdens and projective distortions across two views. Then, inspired by camera paths in video stabilization, we derive the mathematical expression of stitching trajectories in video stitching by elaborately integrating spatial and temporal warps. Finally, a warp smoothing model is presented to produce stable stitched videos with a hybrid loss to simultaneously encourage content alignment, trajectory smoothness, and online collaboration. Compared with StabStitch that sacrifices alignment for stabilization, StabStitch++ makes no compromise and optimizes both of them simultaneously, especially in the online mode. To establish an evaluation benchmark and train the learning framework, we build a video stitching dataset with a rich diversity in camera motions and scenes. Experiments exhibit that StabStitch++ surpasses current solutions in stitching performance, robustness, and efficiency, offering compelling advancements in this field by building a real-time online video stitching system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。