arXiv:2601.00694cs.AI2026-01被引 4

用视觉与交通知识增强大模型,让机器像人一样判断过马路行为。

A Vision-and-Knowledge Enhanced Large Language Model for Generalizable Pedestrian Crossing Behavior Inference

  • 融合视觉特征与交通知识,用低秩微调大模型推理过街行为。
  • 在未见地点上达到66.9%准确率,比传统方法高18个百分点。
  • 仅需5个样例即可实现72.2%准确率,适合新场景快速部署。

现有行人过街行为推断方法(从统计模型到监督学习)泛化能力差,在新地点表现不佳。大语言模型虽具备语义推理优势,但缺乏领域适配与视觉上下文。本文提出视觉-知识增强的行人过街大模型(PedX-LLM),将LLaVA提取的视觉特征与文本数据、交通领域知识结合,通过低秩适应(LoRA)微调LLaMA-2-7B模型以推断过街决策。该模型达82.0%平衡准确率,优于最优统计与监督学习方法。视觉模块提升2.9%,知识融合再增4.1%。跨站点验证显示,零样本配置在五个未见站点上达66.9%平衡准确率,较基线至少高18个百分点;仅用五例少样本微调后升至72.2%。结果表明,视觉与知识增强推理使模型具备类人决策能力,有效突破纯数据驱动方法的局限。

原文摘要 · Abstract (English)

Existing paradigms for inferring pedestrian crossing behavior, ranging from statistical models to supervised learning methods, demonstrate limited generalizability and perform inadequately on new sites. Recent advances in Large Language Models (LLMs) offer a shift from numerical pattern fitting to semantic, context-aware behavioral reasoning, yet existing LLM applications lack domain-specific adaptation and visual context. This study introduces Pedestrian Crossing LLM (PedX-LLM), a vision-and-knowledge enhanced framework designed to transform pedestrian crossing inference from site-specific pattern recognition to generalizable behavioral reasoning. By integrating LLaVA-extracted visual features with textual data and transportation domain knowledge, PedX-LLM fine-tunes a LLaMA-2-7B foundation model via Low-Rank Adaptation (LoRA) to infer crossing decisions. PedX-LLM achieves 82.0% balanced accuracy, outperforming the best statistical and supervised learning methods. Results demonstrate that the vision-augmented module contributes a 2.9% performance gain by capturing the built environment and integrating domain knowledge yields an additional 4.1% improvement. To evaluate generalizability across unseen environments, cross-site validation was conducted using site-based partitioning. The zero-shot PedX-LLM configuration achieves 66.9% balanced accuracy on five unseen test sites, outperforming the baseline data-driven methods by at least 18 percentage points. Incorporating just five validation examples via few-shot learning to PedX-LLM further elevates the balanced accuracy to 72.2%. PedX-LLM demonstrates strong generalizability to unseen scenarios, confirming that vision-and-knowledge-enhanced reasoning enables the model to mimic human-like decision logic and overcome the limitations of purely data-driven methods.

行人行为大模型视觉推理少样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。