arXiv:2602.10458cs.AIcs.LG2026-02被引 3

用大模型提升自动驾驶强化学习,实现近似大模型性能的实时推理。

Found-RL: foundation model-enhanced reinforcement learning for autonomous driving

  • 异步批量推理解耦大模型计算,突破高延迟瓶颈。
  • 通过价值-边际正则与优势加权引导,高效吸收大模型动作建议。
  • 适合追求高性能、低延迟自动驾驶系统的研发团队。

强化学习(RL)已成为端到端自动驾驶(AD)的主流范式,但存在样本效率低和复杂场景下语义可解释性差的问题。基础模型(特别是视觉语言模型,VLM)可通过提供丰富的上下文感知知识缓解此问题,但其高推理延迟阻碍了在高频RL训练循环中的部署。为此,我们提出Found-RL,一个专为利用基础模型增强自动驾驶强化学习而设计的平台。核心创新是异步批量推理框架,将重型VLM推理从仿真循环中解耦,有效解决延迟瓶颈,支持实时学习。引入多种监督机制:值-边际正则化(VMR)和优势加权动作引导(AWAG),以高效将类专家的VLM动作建议融入RL策略。同时采用高吞吐量CLIP进行密集奖励塑造。针对CLIP的动态盲区,提出条件对比动作对齐,基于离散速度/指令条件化提示,实现基于上下文的锚点评分归一化奖励。Found-RL提供微调后VLM集成的端到端流程,证明轻量级RL模型可达到接近百亿参数VLM的性能,且保持实时推理(约500 FPS)。代码、数据与模型将公开于https://github.com/ys-qu/found-rl。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) has emerged as a dominant paradigm for end-to-end autonomous driving (AD). However, RL suffers from sample inefficiency and a lack of semantic interpretability in complex scenarios. Foundation Models, particularly Vision-Language Models (VLMs), can mitigate this by offering rich, context-aware knowledge, yet their high inference latency hinders deployment in high-frequency RL training loops. To bridge this gap, we present Found-RL, a platform tailored to efficiently enhance RL for AD using foundation models. A core innovation is the asynchronous batch inference framework, which decouples heavy VLM reasoning from the simulation loop, effectively resolving latency bottlenecks to support real-time learning. We introduce diverse supervision mechanisms: Value-Margin Regularization (VMR) and Advantage-Weighted Action Guidance (AWAG) to effectively distill expert-like VLM action suggestions into the RL policy. Additionally, we adopt high-throughput CLIP for dense reward shaping. We address CLIP's dynamic blindness via Conditional Contrastive Action Alignment, which conditions prompts on discretized speed/command and yields a normalized, margin-based bonus from context-specific action-anchor scoring. Found-RL provides an end-to-end pipeline for fine-tuned VLM integration and shows that a lightweight RL model can achieve near-VLM performance compared with billion-parameter VLMs while sustaining real-time inference (approx. 500 FPS). Code, data, and models will be publicly available at https://github.com/ys-qu/found-rl.

强化学习自动驾驶大模型实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。