通过动态筛选关键视觉与3D目标信息，让视觉变压器更快实现高精度3D检测。
SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection
- 基于2D图像与3D目标的相关性，联合筛选关键令牌和查询
- 在nuScenes上推理速度提升3倍，准确率损失极小
- 适合需要实时3D感知的自动驾驶系统
视觉变换器（ViTs）在多视角3D检测中表现强劲，但因跨多个视角和大3D区域进行密集令牌与查询处理，导致推理延迟高。现有稀疏化方法主要针对2D视觉，仅裁剪或合并图像令牌，无法实现全模型稀疏或处理3D目标查询。我们提出SToRe3D，一种相关性对齐的稀疏框架，可联合选择2D图像令牌和3D目标查询，并存储过滤后的特征以供重用。双向2D-3D相关性头将计算资源分配给关键内容，同时保留其他嵌入。在nuScenes及我们新构建的nuScenes-Relevance基准上评估，SToRe3D实现了最高3倍的推理加速，准确率损失微小，首次实现基于ViT的大规模实时3D检测，同时保持对规划关键目标的精度。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) enable strong multi-view 3D detection but are limited by high inference latency from dense token and query processing across multiple views and large 3D regions. Existing sparsity methods, designed mainly for 2D vision, prune or merge image tokens but do not extend to full-model sparsity or address 3D object queries. We introduce SToRe3D, a relevance-aligned sparsity framework that jointly selects 2D image tokens and 3D object queries while storing filtered features for reactivation. Mutual 2D-3D relevance heads allocate compute to driving-critical content and preserve other embeddings. Evaluated on nuScenes and our new nuScenes-Relevance benchmark, SToRe3D achieves up to 3x faster inference with marginal accuracy loss, establishing real-time large-scale ViT-based 3D detection while maintaining accuracy on planning-critical agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。