arXiv:2501.13072cs.ROcs.AI2025-01ICLR被引 23

针对自动驾驶世界模型微调时性能下降问题,提出自适应优化方法。

AdaWM: Adaptive World Model based Planning for Autonomous Driving

  • 通过量化策略与模型错配,动态选择微调策略
  • 采用低秩更新实现高效且精准的模型或策略调整
  • 在CARLA上显著提升微调后系统的鲁棒性与效率

基于世界模型的强化学习为自动驾驶提供了有前景的解决方案,通过学习潜在动态模型并用于规划策略训练。为加速学习过程,常采用预训练-微调范式:在线强化学习由离线预训练模型初始化。然而,直接在此基础上进行在线交互可能导致性能大幅下降。我们分析发现,性能下降主要源于策略与动态模型之间的分布偏移所导致的错配。进一步研究表明,微调策略的选择在缓解此类问题中起关键作用。为此,提出AdaWM方法,包含两个核心步骤:(a) 错配识别,量化偏差并指导微调策略;(b) 驱动对齐的微调,根据需要选择性地使用高效的低秩更新调整策略或模型。在挑战性的CARLA驾驶任务上的大量实验表明,AdaWM显著改善了微调过程,使自动驾驶系统具备更强的鲁棒性与更高的效率。

原文摘要 · Abstract (English)

World model based reinforcement learning (RL) has emerged as a promising approach for autonomous driving, which learns a latent dynamics model and uses it to train a planning policy. To speed up the learning process, the pretrain-finetune paradigm is often used, where online RL is initialized by a pretrained model and a policy learned offline. However, naively performing such initialization in RL may result in dramatic performance degradation during the online interactions in the new task. To tackle this challenge, we first analyze the performance degradation and identify two primary root causes therein: the mismatch of the planning policy and the mismatch of the dynamics model, due to distribution shift. We further analyze the effects of these factors on performance degradation during finetuning, and our findings reveal that the choice of finetuning strategies plays a pivotal role in mitigating these effects. We then introduce AdaWM, an Adaptive World Model based planning method, featuring two key steps: (a) mismatch identification, which quantifies the mismatches and informs the finetuning strategy, and (b) alignment-driven finetuning, which selectively updates either the policy or the model as needed using efficient low-rank updates. Extensive experiments on the challenging CARLA driving tasks demonstrate that AdaWM significantly improves the finetuning process, resulting in more robust and efficient performance in autonomous driving systems.

自动驾驶强化学习世界模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。