轻量级适配器让3D模型在鱼眼相机下准确估计深度与位姿。
RayTun3R: Online Camera Adaptation in 3D Foundation Models

- 仅调整位置编码和坐标映射,固定主干网络
- 鱼眼图像旋转误差降低2-12倍,参数仅1万
- 适合需快速部署的机器人、自动驾驶场景
近期3D基础模型如DUSt3R、MASt3R、VGGT、$π^3$和Depth Anything 3在针孔相机图像上表现良好,但在鱼眼相机几何下性能急剧下降。我们发现这是由于预训练模型的位置编码存在针孔相机偏差。为此提出RayTun3R,一种轻量级相机适配方法:保持预训练网络不变,仅优化与标记位置和相机几何相关的轻量组件。该方法学习参数高效的残差修正,作用于绝对与旋转变换位置编码,并通过无参的标记化及预测网格坐标修正,消除残留的针孔假设。适配器仅含10,752个可训练参数,可基于短时序列用几何损失快速训练。适配后能有效迁移至序列其余帧,不增加运行开销。在视场角110°至200°的多种鱼眼数据集上,相比未适配模型,旋转误差降低2–12倍;优于LoRA但参数量少约14倍;在姿态估计上超越无适配基线,且避免多视角推理成本,同时保持深度精度竞争力。
原文摘要 · Abstract (English)
Recent 3D foundation models, such as DUSt3R, MASt3R, VGGT, $π^3$, and Depth Anything 3, provide strong feed-forward depth and pose estimates on pinhole imagery, but degrade sharply under fisheye camera geometry. We show that this failure is partly caused by a pinhole camera bias in the positional encodings of pretrained 3D foundation models, and propose RayTun3R, a lightweight camera adaptation approach. It keeps the pretrained network fixed and adapts only lightweight components tied to token position and camera geometry. RayTun3R learns parameter-efficient residual corrections to absolute and rotary positional encodings, together with parameter-free tokenization and corrections to prediction-grid coordinates that remove residual pinhole assumptions. The resulting adapter contains only 10,752 trainable parameters and can be learned from a short temporal segment using geometric losses. Once adapted, RayTun3R transfers effectively to the remaining frames of the sequence without incurring additional runtime costs. Across diverse fisheye datasets with fields of view from $110^\circ$ to $200^\circ$, our adapter reduces rotation error by $2$-$12\times$ relative to the unadapted model, outperforms LoRA while using $\sim\!14\times$ fewer trainable parameters, improves pose over adaptation-free baselines while avoiding their multi-view inference cost, and remains competitive on depth accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。