用多传感器数据生成可解释的自动驾驶行为描述，提升系统透明度与安全性。
CommandLM: Data driven behavior level descriptor for ego vehicles
- 融合激光雷达与摄像头数据，通过语言模型生成车辆行为摘要。
- 在nuScenes数据集上达到CIDEr 0.67、BERT-F1 0.88，优于基线模型。
- 生成内容可读性强，适合用于规划监督与安全审计，适合开发者与监管者使用。
随着自动驾驶系统向真实世界部署推进，可解释的行为级决策对安全、信任和合规至关重要。我们提出CommandLM，一种多模态大语言模型，能从融合的多传感器数据中生成简洁、人类可读的本车行为描述。该模型通过连接量化且采用LoRA微调的大语言模型的Q-Former适配器，处理来自激光雷达与多相机输入的时间融合鸟瞰图表示。在自建的CommandLM-nuScenes数据集上训练后，CommandLM生成具备意图感知能力、可解释的描述，适用于规划器监督与安全审计。实验表明其语言与行为对齐性优异，取得CIDEr 0.67与BERT-F1 0.88，显著优于BLIP-2基线（CIDEr 0.52，BERT-F1 0.86）。人工评估显示，58%的生成描述被评定为准确、高效且符合规则，证实其现实合理性。尽管部分描述未始终选择最高效的路径行为，但其可解释输出使下游验证系统可识别并纠正此类问题，使其成为透明行为审计的有效工具。结果表明，多模态融合与语言推理结合可实现高效透明的行为级理解。代码与数据集已开源：https://github.com/b-tok/CommandLM
原文摘要 · Abstract (English)
As autonomous driving systems move toward real-world deployment, interpretable, behavior-level decision-making is essential for safety, trust, and regulation. We introduce CommandLM, a multimodal large language model that generates concise, human-readable behavior descriptions for ego vehicles from fused multi-sensor data. Our model processes temporally fused bird's-eye view representations from LiDAR and multi-camera inputs via a Q-Former adapter connected to a quantized, LoRA-fine-tuned large language model. Trained on our CommandLM-nuScenes dataset, CommandLM produces intent-aware, interpretable captions suitable for planner supervision and safety auditing. Experiments demonstrate strong linguistic and behavioral alignment, achieving CIDEr 0.67, and BERT-F1 0.88, substantially outperforming the BLIP-2 baseline (CIDEr 0.52, BERT-F1 0.86). In human evaluation, 58% of the generated descriptions were rated accurate, efficient and rule-compliant, confirming their real-world plausibility. While the remaining descriptions may not always select the most efficient, goal-oriented behavior, CommandLM's interpretable outputs enable downstream validation systems to identify and correct such cases, making it an effective tool for transparent behavior auditing. These results show that integrating multimodal fusion with language reasoning yields efficient and transparent behavior-level understanding for autonomous driving. We release our code and dataset at: https://github.com/b-tok/CommandLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。