用3D高斯泼溅实现物理AI的视点扩展，提升新视角下深度预测稳定性。
Splat2Real: Novel-view Scaling for Physical AI with 3D Gaussian Splatting
- 以数字孪生为教师，通过视点覆盖与新颖性联合优化选择训练视图。
- 在20个TUM数据集上，新增视图数达2000时仍保持低误差，优于传统策略。
- 适合需要鲁棒新视角感知的机器人视觉与自主系统研究者。
物理AI面临训练与部署阶段视角差异问题，新视角鲁棒性对单目RGB转3D感知至关重要。本文将Real2Render2Real单目深度预训练视为从数字孪生代理中学习的模仿学习：学生深度网络模仿由场景网格渲染出的专家度量深度/可见性，而3DGS提供可扩展的新视角观测。我们提出Splat2Real，聚焦新视角扩展：性能更依赖于添加视点的质量而非数量。引入CN-Coverage——一种结合覆盖度与新颖性的课程学习策略，通过几何增益与外推惩罚贪婪选择视点，并设置质量感知的容错机制应对低可靠性教师。在20个TUM RGB-D序列上，步匹配预算（N=0至2000个附加渲染视图，唯一视图≤500，大预算下重采样），朴素扩展不稳定；CN-Coverage缓解最差情况退化，相比Robot/Coverage策略表现更优；GOL-Gated CN-Coverage在中高预算下实现最强稳定性，且高新颖性尾部误差最低。下游控制代理结果随N变化，验证了其在视角偏移下的具身相关性，体现安全与进展权衡的动态转移。
原文摘要 · Abstract (English)
Physical AI faces viewpoint shift between training and deployment, and novel-view robustness is essential for monocular RGB-to-3D perception. We cast Real2Render2Real monocular depth pretraining as imitation-learning-style supervision from a digital twin oracle: a student depth network imitates expert metric depth/visibility rendered from a scene mesh, while 3DGS supplies scalable novel-view observations. We present Splat2Real, centered on novel-view scaling: performance depends more on which views are added than on raw view count. We introduce CN-Coverage, a coverage+novelty curriculum that greedily selects views by geometry gain and an extrapolation penalty, plus a quality-aware guardrail fallback for low-reliability teachers. Across 20 TUM RGB-D sequences with step-matched budgets (N=0 to 2000 additional rendered views, with N unique <= 500 and resampling for larger budgets), naive scaling is unstable; CN-Coverage mitigates worst-case regressions relative to Robot/Coverage policies, and GOL-Gated CN-Coverage provides the strongest medium-high-budget stability with the lowest high-novelty tail error. Downstream control-proxy results versus N provides embodied-relevance evidence by shifting safety/progress trade-offs under viewpoint shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。