arXiv:2510.22370cs.ROcs.AI2025-10

用视觉语言模型融合车道保持的语义与几何信息,提升自动驾驶稳定性。

BLIP-FusePPO: A Vision-Language Deep Reinforcement Learning Framework for Lane Keeping in Autonomous Vehicles

  • 将视觉语言模型的语义嵌入直接融入状态表示,结合几何与控制信号。
  • 在复杂场景下比现有最优方法更稳定,车道保持准确率显著提升。
  • 无需运行时额外推理,适合实时自动驾驶系统部署。

本文提出一种基于自举视觉-语言预训练的融合状态表示近端策略优化框架(BLIP-FusePPO),用于自动驾驶车道保持任务。该框架将视觉语言模型生成的语义嵌入直接融合至智能体观测空间中的几何状态、激光雷达数据及比例积分微分(PID)控制反馈中。通过结合高层场景理解与低层控制信号,使策略学习更鲁棒。采用包含语义对齐、车道保持精度、障碍物避让和速度调节的混合奖励函数,提升学习效率与泛化能力。与仅用语义模型设计奖励的方法不同,本方法直接将语义特征嵌入状态表示,避免昂贵的运行时推理,确保语义引导始终可用。仿真结果表明,该模型在多种复杂驾驶场景下均优于当前最佳的视觉与多模态强化学习基线,在车道保持稳定性与适应性方面表现更优。代码已开源。

原文摘要 · Abstract (English)

In this paper, we propose Bootstrapped Language-Image Pretraining-driven Fused State Representation in Proximal Policy Optimization (BLIP-FusePPO), a novel multimodal reinforcement learning (RL) framework for autonomous lane-keeping (LK), in which semantic embeddings generated by a vision-language model (VLM) are directly fused with geometric states, LiDAR observations, and Proportional-Integral-Derivative-based (PID) control feedback within the agent observation space. The proposed method lets the agent learn driving rules that are aware of their surroundings and easy to understand by combining high-level scene understanding from the VLM with low-level control and spatial signals. Our architecture brings together semantic, geometric, and control-aware representations to make policy learning more robust. A hybrid reward function that includes semantic alignment, LK accuracy, obstacle avoidance, and speed regulation helps learning to be more efficient and generalizable. Our method is different from the approaches that only use semantic models to shape rewards. Instead, it directly embeds semantic features into the state representation. This cuts down on expensive runtime inference and makes sure that semantic guidance is always available. The simulation results show that the proposed model is better at LK stability and adaptability than the best vision-based and multimodal RL baselines in a wide range of difficult driving situations. We make our code publicly available.

自动驾驶强化学习多模态车道保持

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。