让快速3D重建模型输出更易优化的初始结果,提升质量且无需增加推理开销。
ForeSplat: Optimization-Aware Foresight for Feed-Forward 3D Gaussian Splatting

- 训练时考虑下游优化器,使预测结果更适合快速精修。
- 用少量优化步数达到更高重建质量,甚至超过全优化结果。
- 轻量级设计适合边缘设备,兼容多种模型结构。
前馈式3D高斯点云(3DGS)模型可实现快速单遍重建,但其质量受限于大规模3D标注数据的稀缺性。现有方法采用‘预测后精修’策略,由后续优化弥补前馈网络容量不足。然而,标准3DGS仅针对零步渲染误差训练,未考虑输出是否适合作为优化器的良好初始化。本文提出ForeSplat,一种面向优化的训练框架,使前馈3DGS生成专为快速高效精修设计的初始状态。通过将部分建模任务交由优化器承担,显著降低前馈模型容量压力,即使使用紧凑网络也能实现高质量重建。核心是MetaGrad——一种轻量级多锚点元梯度训练规则,通过短内循环精修轨迹采样锚点状态,并回传聚合的一阶梯度作为优化感知信号,避免高阶微分开销。该微调无推理成本,仅需数步精修即可实现秒级高质量重建。我们在AnySplat、Pi3X及面向边缘部署的蒸馏版模型上验证,所有架构中,ForeSplat训练的初始化均以更少迭代次数收敛,且峰值重建质量超越原始模型,甚至优于完全收敛的基线。该框架持续缩小渐进预测与逐场景优化之间的差距,为轻量级高保真3D重建提供可行路径。
原文摘要 · Abstract (English)
Feed-forward 3D Gaussian Splatting models offer fast single-pass reconstruction,but scaling them to match per-scene optimization quality is fundamentally hindered by the scarcity of large-scale 3D annotations. A practical compromise is predict-then-refine,where post-prediction optimization compensates for the limited capacity of the feed-forward network. However,standard feed-forward 3DGS is trained solely for zero-step rendering error,ignoring whether its output constitutes a good initialization for the downstream optimizer. We present ForeSplat,an optimization-aware training framework that equips feed-forward 3DGS models to produce initializations explicitly designed for rapid,effective refinement. By offloading part of the scene-modeling burden to the optimizer,ForeSplat substantially reduces the capacity pressure on the feed-forward model,making high-quality reconstruction feasible even with compact networks. At its core is MetaGrad,a lightweight multi-anchor meta-gradient training rule that bypasses costly higher-order differentiation through the 3DGS optimizer. MetaGrad unrolls a short inner-loop refinement trajectory,samples anchor states,and back-propagates aggregated first-order gradients to the prediction head as a surrogate optimization-aware signal. This fine-tuning adds no inference cost and enables high-quality reconstruction within seconds after a few refinement steps. We instantiate ForeSplat on diverse backbones,including AnySplat,Pi3X,and a distilled variant tailored for edge deployment. Across all tested architectures,a ForeSplat-trained initialization converges in fewer refinement steps and reaches a higher peak reconstruction quality than its vanilla counterpart,even fully converged. The framework consistently bridges the gap between amortized prediction and per-scene optimization,establishing a practical path toward lightweight,high-fidelity 3D reconstruction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。