测试时用先验约束优化模型，无需重训练即可提升3D重建效果。
Learning 3D Reconstruction with Priors in Test Time
- 在推理时引入相机位姿、内参等先验作为约束条件，不修改网络结构
- 在ETH3D等数据集上点图误差降低超过50% ，显著优于基础模型
- 适用于已有图像模型但需融合先验的3D视觉任务，尤其适合部署场景
我们提出一种测试时框架，用于多视图Transformer（MVTs）在不重新训练或修改预训练图像模型的前提下，利用先验信息（如相机位姿、内参、深度）提升3D任务表现。不同于将先验直接输入网络，我们将其转化为对预测结果的约束，并在推理时通过优化实现。损失函数包含自监督项和先验惩罚项：自监督项通过多视角渲染间的光度或几何一致性来衡量预测兼容性；可用先验则转化为对应输出模态的惩罚项。在多个3D视觉基准测试中，包括点图估计和相机位姿估计，该方法显著优于基线MVT模型。在ETH3D、7-Scenes和NRGBD数据集上，点图距离误差减少超过一半。同时优于重新训练的前向式先验感知方法，验证了测试时约束优化（TCO）框架的有效性。
原文摘要 · Abstract (English)
We introduce a test-time framework for multiview Transformers (MVTs) that incorporates priors (e.g., camera poses, intrinsics, and depth) to improve 3D tasks without retraining or modifying pre-trained image-only networks. Rather than feeding priors into the architecture, we cast them as constraints on the predictions and optimize the network at inference time. The optimization loss consists of a self-supervised objective and prior penalty terms. The self-supervised objective captures the compatibility among multi-view predictions and is implemented using photometric or geometric loss between renderings from other views and each view itself. Any available priors are converted into penalty terms on the corresponding output modalities. Across a series of 3D vision benchmarks, including point map estimation and camera pose estimation, our method consistently improves performance over base MVTs by a large margin. On the ETH3D, 7-Scenes, and NRGBD datasets, our method reduces the point-map distance error by more than half compared with the base image-only models. Our method also outperforms retrained prior-aware feed-forward methods, demonstrating the effectiveness of our test-time constrained optimization (TCO) framework for incorporating priors into 3D vision tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。