单目相机下高效完成3D语义场景补全，同时输出预测置信度。
ET-Former: Efficient Triplane Deformable Attention for 3D Semantic Scene Completion From Monocular Camera
- 基于三平面可变形注意力，提升几何理解与语义精度。
- 在Semantic-KITTI上实现51.49的IoU和16.30的mIoU，刷新纪录。
- 低内存占用（仅10.9GB），适合部署于资源受限设备。
我们提出ET-Former，一种基于单目相机的端到端语义场景补全算法。该方法从单张RGB图像生成语义占据图，并同时提供语义预测的不确定性估计。通过设计三平面结构的可变形注意力机制，模型在场景几何理解上优于现有SOTA方法，有效降低语义预测噪声。此外，利用条件变分自编码器（CVAE）对预测不确定性进行建模。生成的语义图与不确定性图可支持安全导航策略制定。在Semantic-KITTI数据集上的实验表明，ET-Former在测试集上将IoU从44.71提升至51.49，mIoU从15.04提升至16.30，同时保持最低的GPU内存消耗，训练仅需10.9 GB。项目主页：https://github.com/jingGM/ET-Former.git。
原文摘要 · Abstract (English)
We introduce ET-Former, a novel end-to-end algorithm for semantic scene completion using a single monocular camera. Our approach generates a semantic occupancy map from single RGB observation while simultaneously providing uncertainty estimates for semantic predictions. By designing a triplane-based deformable attention mechanism, our approach improves geometric understanding of the scene than other SOTA approaches and reduces noise in semantic predictions. Additionally, through the use of a Conditional Variational AutoEncoder (CVAE), we estimate the uncertainties of these predictions. The generated semantic and uncertainty maps will help formulate navigation strategies that facilitate safe and permissible decision making in the future. Evaluated on the Semantic-KITTI dataset, ET-Former achieves the highest Intersection over Union (IoU) and mean IoU (mIoU) scores while maintaining the lowest GPU memory usage, surpassing state-of-the-art (SOTA) methods. It improves the SOTA scores of IoU from 44.71 to 51.49 and mIoU from 15.04 to 16.30 on SeamnticKITTI test, with a notably low training memory consumption of 10.9 GB. Project page: https://github.com/jingGM/ET-Former.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。