构建首个高速公路多模态大模型,实现事件识别与应急响应的智能决策。
ExpressMind: A Multimodal Pretrained Large Language Model for Expressway Operation
- 基于自监督与无监督学习的双层预训练,融合文本、视频与知识图谱。
- 在多模态高速基准上全面超越基线,在事件检测与安全响应生成上提升显著。
- 适合交通智能系统研发者与自动驾驶场景理解研究者使用。
当前高速公路运营依赖规则驱动且孤立的模型,难以跨系统协同分析知识。尽管大语言模型正推动交通系统从算法智能迈向认知智能,但通用模型难以理解高速公路非常规场景中的法规与因果关系。为此,本文构建首个面向高速公路的多模态预训练大语言模型ExpressMind,作为智能高速运营的认知核心。首次建立全栈高速公路数据集,涵盖交通知识文本、应急推理链与标注视频事件,缓解数据稀缺问题。提出基于自监督与无监督学习的双层预训练范式,并引入图增强RAG框架动态索引高速知识库。为提升事故响应策略推理能力,设计强化学习对齐的思维链机制(RL-CoT),确保模型推理与专家处理逻辑一致。最后,集成跨模态编码器对齐视觉与文本通道的动态特征序列,实现对视频与图像模态交通场景的理解。在新发布的多模态高速公路基准上,实验表明ExpressMind在事件检测、安全响应生成与复杂交通分析上全面优于现有基线。代码与数据已开源:https://wanderhee.github.io/ExpressMind/
原文摘要 · Abstract (English)
The current expressway operation relies on rule-based and isolated models, which limits the ability to jointly analyze knowledge across different systems. Meanwhile, Large Language Models (LLMs) are increasingly applied in intelligent transportation, advancing traffic models from algorithmic to cognitive intelligence. However, general LLMs are unable to effectively understand the regulations and causal relationships of events in unconventional scenarios in the expressway field. Therefore, this paper constructs a pre-trained multimodal large language model (MLLM) for expressways, ExpressMind, which serves as the cognitive core for intelligent expressway operations. This paper constructs the industry's first full-stack expressway dataset, encompassing traffic knowledge texts, emergency reasoning chains, and annotated video events to overcome data scarcity. This paper proposes a dual-layer LLM pre-training paradigm based on self-supervised training and unsupervised learning. Additionally, this study introduces a Graph-Augmented RAG framework to dynamically index the expressway knowledge base. To enhance reasoning for expressway incident response strategies, we develop a RL-aligned Chain-of-Thought (RL-CoT) mechanism that enforces consistency between model reasoning and expert problem-solving heuristics for incident handling. Finally, ExpressMind integrates a cross-modal encoder to align the dynamic feature sequences under the visual and textual channels, enabling it to understand traffic scenes in both video and image modalities. Extensive experiments on our newly released multi-modal expressway benchmark demonstrate that ExpressMind comprehensively outperforms existing baselines in event detection, safety response generation, and complex traffic analysis. The code and data are available at: https://wanderhee.github.io/ExpressMind/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。