让自动驾驶画面生成突破固定路线限制，自由生成任意新视角。
FreeVS: Generative View Synthesis on Free Driving Trajectory
- 用伪图像表征控制生成，实现3D一致性与姿态准确性。
- 在Waymo数据集上，新轨迹合成图像质量显著优于现有方法。
- 专为自由视角设计新评测基准，适合真实驾驶场景研究者。
现有基于重建的驾驶场景新视角合成方法仅限于车辆原路径视角生成，偏离路径时渲染效果急剧下降。本文提出FreeVS，一种完全生成式新视角合成方法，可在真实驾驶场景中自由生成任意新轨迹的视角。通过伪图像表征视点先验，模拟相机在各方向移动，确保生成结果与真实场景三维一致且姿态准确。训练完成后，无需重建过程即可在任意验证序列上生成新视角。此外，提出两个新挑战性基准：新视角合成与新轨迹合成，强调视角自由度。由于新轨迹无真值图像，引入3D感知模型评估合成图像一致性。在Waymo Open Dataset上的实验表明，FreeVS在原路径与新路径上均具备优异图像合成性能。
原文摘要 · Abstract (English)
Existing reconstruction-based novel view synthesis methods for driving scenes focus on synthesizing camera views along the recorded trajectory of the ego vehicle. Their image rendering performance will severely degrade on viewpoints falling out of the recorded trajectory, where camera rays are untrained. We propose FreeVS, a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes. To control the generation results to be 3D consistent with the real scenes and accurate in viewpoint pose, we propose the pseudo-image representation of view priors to control the generation process. Viewpoint transformation simulation is applied on pseudo-images to simulate camera movement in each direction. Once trained, FreeVS can be applied to any validation sequences without reconstruction process and synthesis views on novel trajectories. Moreover, we propose two new challenging benchmarks tailored to driving scenes, which are novel camera synthesis and novel trajectory synthesis, emphasizing the freedom of viewpoints. Given that no ground truth images are available on novel trajectories, we also propose to evaluate the consistency of images synthesized on novel trajectories with 3D perception models. Experiments on the Waymo Open Dataset show that FreeVS has a strong image synthesis performance on both the recorded trajectories and novel trajectories. Project Page: https://freevs24.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。