无需已知相机参数，实时生成多视角新视图视频。
LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video
- 用视觉变换器+扩散插值实现无标定多视角实时重建
- 每帧仅需0.07秒，速度比传统方法快数十倍
- 适合低延迟直播场景，仅需两路同步输入
从无标定的稀疏多视角视频中实时进行新视图合成（NVS）仍是广泛应用场景中的开放挑战。现有动态场景建模方法通常依赖真实相机参数，并涉及长时间优化（约2.67秒），难以满足直播需求。为此，我们提出一种新型前馈式视点视频直播方法LiveStre4m，可实现实时从无标定稀疏多视角输入中生成新视图。LiveStre4m采用多视角视觉变换器进行关键帧3D场景重建，结合扩散-变换器插值模块以保证时间一致性与稳定流输出。同时，设计了相机位姿预测模块，直接从RGB图像估计相机位姿与内参，摆脱对已知标定信息的依赖。该方法仅需两路同步的未标定输入流即可实现时间一致的新视图视频直播。在1024×768分辨率下，每帧平均重建耗时仅0.07秒，相比基于优化的动态场景表示方法在运行时间上提升数个数量级。结果表明，LiveStre4m使实时NVS直播在实际场景中成为可能，标志着可部署直播新视图合成系统的重要进展。代码已公开：https://github.com/pedro-quesado/LiveStre4m
原文摘要 · Abstract (English)
Live-streaming Novel View Synthesis (NVS) from unposed multi-view video remains an open challenge in a wide range of applications. Existing methods for dynamic scene representation typically require ground-truth camera parameters and involve lengthy optimizations ($\approx 2.67$s), which makes them unsuitable for live streaming scenarios. To address this issue, we propose a novel viewpoint video live-streaming method (LiveStre4m), a feed-forward model for real-time NVS from unposed sparse multi-view inputs. LiveStre4m introduces a multi-view vision transformer for keyframe 3D scene reconstruction coupled with a diffusion-transformer interpolation module that ensures temporal consistency and stable streaming. In addition, a Camera Pose Predictor module is proposed to efficiently estimate both poses and intrinsics directly from RGB images, removing the reliance on known camera calibration information. Our approach enables temporally consistent novel-view video streaming in real-time using as few as two synchronized unposed input streams. LiveStre4m attains an average reconstruction time of $ 0.07$s per-frame at $ 1024 \times 768$ resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https://github.com/pedro-quesado/LiveStre4m
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。