给大模型驾驶系统加了个红绿灯识别模块，让决策更安全可靠。
Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition
- 用模块化结构将检测结果转为自然语言注入大模型输入
- 在CARLA测试中驾驶表现提升最高14%，违规减少且稳定性更强
- 无需改动主模型，适配单/多摄像头，适合追求安全的自动驾驶研究者
大型语言模型（LLMs）在自动驾驶决策与规划中日益广泛应用，展现出出色的推理能力及跨复杂交通场景的泛化潜力。然而，现有基于LLM的驾驶代理缺乏显式交通规则执行机制，难以可靠检测红绿灯、标志等小型关键物体。为此，我们提出TLS-Assist——一种模块化冗余层，增强基于LLM的自动驾驶系统对红绿灯和交通标志的识别能力。该框架将检测结果转化为结构化自然语言信息并注入到LLM输入中，强制模型关注安全关键线索。其设计为即插即用、与模型无关，支持单目与多视角相机配置。我们在CARLA环境下的LangAuto基准进行闭环评估，结果显示相比LMDrive性能提升达14%，相比BEVDriver提升7%，同时显著降低红绿灯与标志违规次数。代码与模型已公开于https://github.com/iis-esslingen/TLS-Assist。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used for decision-making and planning in autonomous driving, showing promising reasoning capabilities and potential to generalize across diverse traffic situations. However, current LLM-based driving agents lack explicit mechanisms to enforce traffic rules and often struggle to reliably detect small, safety-critical objects such as traffic lights and signs. To address this limitation, we introduce TLS-Assist, a modular redundancy layer that augments LLM-based autonomous driving agents with explicit traffic light and sign recognition. TLS-Assist converts detections into structured natural language messages that are injected into the LLM input, enforcing explicit attention to safety-critical cues. The framework is plug-and-play, model-agnostic, and supports both single-view and multi-view camera setups. We evaluate TLS-Assist in a closed-loop setup on the LangAuto benchmark in CARLA. The results demonstrate relative driving performance improvements of up to 14% over LMDrive and 7% over BEVDriver, while consistently reducing traffic light and sign infractions. We publicly release the code and models on https://github.com/iis-esslingen/TLS-Assist.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。