轻量级3D重建框架,提速降存还保精度。
Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction

- 用稀疏线性注意力替代密集注意力,降低计算开销。
- 支持低精度推理,延迟降1.7-2.0倍,显存省1.9-2.4倍。
- 适配多种模型,适合部署于资源受限的3D重建场景。
基于Transformer的3D重建在多视角观测下能恢复几何与外观,性能强大。但随着模型规模和输入分辨率增大,效率问题日益突出。现有3D Transformer管道面临双重挑战:密集多视角注意力带来巨大计算负担,低精度执行会破坏对几何敏感的表示,导致深度、姿态和3D一致性下降。为此,我们提出Lite3R——一种模型无关的师生框架,以稀疏线性注意力替代密集注意力,在保留关键几何交互的同时显著降低注意力开销。同时引入参数高效的FP8感知量化感知训练(FP8-aware QAT)策略,通过部分注意力蒸馏冻结大部分预训练主干参数,仅训练轻量级线性投影层,实现稳定低精度部署并保留预训练几何先验。我们在VGGT和DA3-Large两个骨干网络上,于BlendedMVS和DTU64数据集上评估,结果表明其显著降低延迟(1.7–2.0倍)和内存占用(1.9–2.4倍),整体重建质量保持竞争力。该工作展示了面向实际部署的高效算法-系统协同设计路径。
原文摘要 · Abstract (English)
Transformer-based 3D reconstruction has emerged as a powerful paradigm for recovering geometry and appearance from multi-view observations, offering strong performance across challenging visual conditions. As these models scale to larger backbones and higher-resolution inputs, improving their efficiency becomes increasingly important for practical deployment. However, modern 3D transformer pipelines face two coupled challenges: dense multi-view attention creates substantial token-mixing overhead, and low-precision execution can destabilize geometry-sensitive representations and degrade depth, pose, and 3D consistency. To address the first challenge, we propose Lite3R, a model-agnostic teacher-student framework that replaces dense attention with Sparse Linear Attention to preserve important geometric interactions while reducing attention cost. To address the second challenge, we introduce a parameter-efficient FP8-aware quantization-aware training (FP8-aware QAT) strategy with partial attention distillation, which freezes the vast majority of pretrained backbone parameters and trains only lightweight linear-branch projection layers, enabling stable low-precision deployment while retaining pretrained geometric priors. We further evaluate Lite3R on two representative backbones, VGGT and DA3-Large, over BlendedMVS and DTU64, showing that it substantially reduces latency (1.7-2.0x) and memory usage (1.9-2.4x) while preserving competitive reconstruction quality overall. These results demonstrate that Lite3R provides an effective algorithm-system co-design approach for practical transformer-based 3D reconstruction. Code: https://github.com/AIGeeksGroup/Lite3R. Website: https://aigeeksgroup.github.io/Lite3R.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。