用视觉语言模型解决自动驾驶语言指令任务，获CVPR挑战赛第一名。
The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge
- 基于LLaVA模型，结合LoRA/DoRA微调提升性能。
- 融合开源深度估计信息，提升推理准确率至0.7799。
- 采用思维链推理处理多选与是非题，适合自动驾驶理解任务。
本文介绍了我们在CVPR 2024自动驾驶大赛「驾驶语言指令」赛道中的系统方案。我们仅使用DriveLM-nuScenes数据集训练模型，基于LLaVA架构，通过LoRA和DoRA方法进行微调，并引入开源深度估计模型提供的深度信息以增强训练与推理过程。在推理阶段，针对多选题和是非题，采用思维链（Chain-of-Thought）推理策略，显著提升答案准确性。该综合方法使我们在验证集排行榜上取得0.7799的最高分，位列榜首。
原文摘要 · Abstract (English)
This report outlines our approach using vision language model systems for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge. We have exclusively utilized the DriveLM-nuScenes dataset for training our models. Our systems are built on the LLaVA models, which we enhanced through fine-tuning with the LoRA and DoRA methods. Additionally, we have integrated depth information from open-source depth estimation models to enrich the training and inference processes. For inference, particularly with multiple-choice and yes/no questions, we adopted a Chain-of-Thought reasoning approach to improve the accuracy of the results. This comprehensive methodology enabled us to achieve a top score of 0.7799 on the validation set leaderboard, ranking 1st on the leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。