用视觉语言模型实现交通场景语义理解与风险定位，仅需前视图像。
MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios
- 双分支视觉编码器提取多分辨率特征，提升小目标描述精度。
- 场景理解任务达80.1% BLEU-1与298.5% CIDEr，定位准确率59.6%。
- 适合自动驾驶中端到端的感知与决策研究者使用。
多模态大语言模型（MLLMs）在自动驾驶多个任务中表现优异。本文提出一种仅依赖前视图像的联合语义场景理解与风险定位框架——MLLM-SUL。该框架采用双分支视觉编码器，从两个分辨率提取特征，增强对不同尺寸风险对象的描述能力；随后微调LLaMA模型生成包含驾驶场景类型、风险对象行为及本车意图与建议的场景描述；最后通过含回归标记的Transformer网络实现风险对象定位。在现有DRAMA-ROLISP数据集和扩展版DRAMA-SRIS数据集上的实验表明，该方法高效且优于多项基于图像与视频的先进方法：场景理解任务中达到80.1% BLEU-1与298.5% CIDEr，定位任务准确率达59.6%。代码与数据集已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have shown satisfactory effects in many autonomous driving tasks. In this paper, MLLMs are utilized to solve joint semantic scene understanding and risk localization tasks, while only relying on front-view images. In the proposed MLLM-SUL framework, a dual-branch visual encoder is first designed to extract features from two resolutions, and rich visual information is conducive to the language model describing risk objects of different sizes accurately. Then for the language generation, LLaMA model is fine-tuned to predict scene descriptions, containing the type of driving scenario, actions of risk objects, and driving intentions and suggestions of ego-vehicle. Ultimately, a transformer-based network incorporating a regression token is trained to locate the risk objects. Extensive experiments on the existing DRAMA-ROLISP dataset and the extended DRAMA-SRIS dataset demonstrate that our method is efficient, surpassing many state-of-the-art image-based and video-based methods. Specifically, our method achieves 80.1% BLEU-1 score and 298.5% CIDEr score in the scene understanding task, and 59.6% accuracy in the localization task. Codes and datasets are available at https://github.com/fjq-tongji/MLLM-SUL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。