用重建场景生成高质量新视角数据,解决3D视觉训练数据少的难题。
FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation

- 基于不完美重建场景,设计不确定性感知的新视角采样策略。
- 使前馈式新视角合成模型在分布外测试中PSNR提升2.7 dB。
- 生成数据可优化3D高斯点云,适合3D重建与视觉任务研究者。
通用新视角合成(NVS)模型的发展受限于多样且精确相机轨迹的大规模训练数据稀缺。真实采集数据虽逼真但稀疏离散,合成数据虽易扩展却存在域偏移且缺乏真实语义。本文提出FreeScale框架,利用场景重建能力将有限的真实图像序列转化为可扩展的高质量训练数据源。核心思想是:不完美的重建场景可作为丰富的几何代理,但直接采样会放大伪影。为此,我们提出一种不确定性感知的自由视角采样策略,识别语义有意义且受重建误差影响最小的新视角。实验表明,利用FreeScale扩展前馈NVS模型训练,在具有挑战性的分布外基准上实现2.7 dB的PSNR提升。此外,生成数据还能主动提升单场景3D高斯点云优化效果,在多个数据集上均取得稳定改进。本工作提供了一种实用而强大的数据生成引擎,突破3D视觉中的根本瓶颈。
原文摘要 · Abstract (English)
The development of generalizable Novel View Synthesis (NVS) models is critically limited by the scarcity of large-scale training data featuring diverse and precise camera trajectories. While real-world captures are photorealistic, they are typically sparse and discrete. Conversely, synthetic data scales but suffers from a domain gap and often lacks realistic semantics. We introduce FreeScale, a novel framework that leverages the power of scene reconstruction to transform limited real-world image sequences into a scalable source of high-quality training data. Our key insight is that an imperfect reconstructed scene serves as a rich geometric proxy, but naively sampling from it amplifies artifacts. To this end, we propose a certainty-aware free-view sampling strategy identifying novel viewpoints that are both semantically meaningful and minimally affected by reconstruction errors. We demonstrate FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR on challenging out-of-distribution benchmarks. Furthermore, we show that the generated data can actively enhance per-scene 3D Gaussian Splatting optimization, leading to consistent improvements across multiple datasets. Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision. Project page: https://mvp-ai-lab.github.io/FreeScale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。