通过随机遮蔽图像块,让自动驾驶模型学会忽略冗余信息,提升泛化能力。
See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection
- 随机遮蔽部分图像块,迫使模型依赖不变特征做决策。
- 在多种分布外场景中平均提升6.2%,闭环仿真最高提升20.4%。
- 方法简单高效,训练速度快2.4倍,可直接部署到真实车辆。
近期端到端自动驾驶研究发现,基于基础模型提取的图像块特征训练的策略在分布外(OOD)场景中具有更好泛化性。我们假设由于自注意力机制,每个图像块特征隐式包含其他所有块的信息,导致特征高度冗余。通过主成分分析和跨块相似性量化,发现90%的方差仅由17/64个主成分解释,且块间相关性普遍强。在此基础上,我们提出随机块选择(SPS)方法:每帧随机遮蔽部分块描述符,保留剩余块的空间布局,使策略面对不同但完整的场景视图。由此,策略学习依赖于对具体块是否出现不敏感的特征。大量实验表明,本方法在所有OOD场景下均优于现有最先进水平,平均提升6.2%,闭环仿真最高达20.4%,且训练速度提升2.4倍。我们系统性测试了遮蔽率与特征重组方式,共训练评估9套系统,其中8套超越先前最先进水平。最终证明,同一策略无需微调即可在真实物理车辆上运行。
原文摘要 · Abstract (English)
Recent advances in end-to-end autonomous driving show that policies trained on patch-aligned features extracted from foundation models generalize better to Out-of-Distribution (OOD). We hypothesize that due to the self-attention mechanism, each patch feature implicitly embeds/contains information from all other patches, represented in a different way and intensity, making these descriptors highly redundant. We quantify redundancy in such (BLIP2) features via PCA and cross-patch similarity: $90$% of variance is captured by $17/64$ principal components, and strong inter-token correlations are pervasive. Training on such overlapping information leads the policy to overfit spurious correlations, hurting OOD robustness. We present Stochastic-Patch-Selection (SPS), a simple yet effective approach for learning policies that are more robust, generalizable, and efficient. For every frame, SPS randomly masks a fraction of patch descriptors, not feeding them to the policy model, while preserving the spatial layout of the remaining patches. Thus, the policy is provided with different stochastic but complete views of the (same) scene: every random subset of patches acts like a different, yet still sensible, coherent projection of the world. The policy thus bases its decisions on features that are invariant to which specific tokens survive. Extensive experiments confirm that across all OOD scenarios, our method outperforms the state of the art (SOTA), achieving a $6.2$% average improvement and up to $20.4$% in closed-loop simulations, while being $2.4\times$ faster. We conduct ablations over masking rates and patch-feature reorganization, training and evaluating 9 systems, with 8 of them surpassing prior SOTA. Finally, we show that the same learned policy transfers to a physical, real-world car without any tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。