用开源多模态模型微调，实现语言驱动自动驾驶的顶尖表现
Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024
- 基于InternVL-1.5全参数微调，融合多视角图像输入
- 单模型在竞赛榜单上取得0.6002的高分，领先多数方案
- 设计自动标注策略，高效生成物体边界框，提升训练效率
本文介绍了我们在CVPR 2024自动驾驶大奖赛「语言驱动驾驶」赛道中采用的方法。我们使用强大的开源多模态模型InternVL-1.5，对竞赛数据集DriveLM-nuScenes进行全参数微调。为有效处理nuScenes数据集的多视角图像，并无缝继承InternVL出色的多模态理解能力，我们以特定方式格式化并拼接多视图图像。这使得最终模型在满足竞赛任务需求的同时，充分保留了InternVL的强大图像理解能力。同时，我们设计了一种简单的自动标注策略，将DriveLM-nuScenes中物体的中心点转换为对应的边界框。结果表明，我们的单模型在最终排行榜上取得了0.6002的得分。
原文摘要 · Abstract (English)
This technical report describes the methods we employed for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge. We utilized a powerful open-source multimodal model, InternVL-1.5, and conducted a full-parameter fine-tuning on the competition dataset, DriveLM-nuScenes. To effectively handle the multi-view images of nuScenes and seamlessly inherit InternVL's outstanding multimodal understanding capabilities, we formatted and concatenated the multi-view images in a specific manner. This ensured that the final model could meet the specific requirements of the competition task while leveraging InternVL's powerful image understanding capabilities. Meanwhile, we designed a simple automatic annotation strategy that converts the center points of objects in DriveLM-nuScenes into corresponding bounding boxes. As a result, our single model achieved a score of 0.6002 on the final leadboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。