评测GPT-4与Phi-3-mini在交通规划中的表现，发现前者更可靠。
Beyond Words: Evaluating Large Language Models in Transportation Planning
- 构建交通导向评估框架，测试LLMs的地理空间与专业能力
- GPT-4在多数任务中准确率显著高于Phi-3-mini
- 适合交通规划、城市研究者关注生成式AI的实际应用
2023年生成式人工智能（GenAI）的兴起推动了城市交通与物流等行业的变革。本研究评估大型语言模型（LLMs）GPT-4与Phi-3-mini在交通规划中的表现，采用包含通用地理空间技能、交通领域知识及真实场景问题解决能力的多维度评估框架。通过混合方法，考察模型在一般地理信息系统（GIS）能力、交通领域知识以及拥堵定价等现实交通规划决策支持中的表现。结果显示，相较于Phi-3-mini，GPT-4在各类地理空间与交通特定任务中展现出更高准确性和可靠性，具备成为交通规划有力工具的潜力。然而，Phi-3-mini在特定分析场景中仍具实用性，表明其在资源受限环境下的适用性。研究强调了GenAI技术在城市交通规划中的变革潜力。未来可探索更先进LLMs及检索增强生成（RAG）技术在更广泛交通规划与运营挑战中的应用，以深化先进AI模型在交通管理中的整合。
原文摘要 · Abstract (English)
The resurgence and rapid advancement of Generative Artificial Intelligence (GenAI) in 2023 has catalyzed transformative shifts across numerous industry sectors, including urban transportation and logistics. This study investigates the evaluation of Large Language Models (LLMs), specifically GPT-4 and Phi-3-mini, to enhance transportation planning. The study assesses the performance and spatial comprehension of these models through a transportation-informed evaluation framework that includes general geospatial skills, general transportation domain skills, and real-world transportation problem-solving. Utilizing a mixed-methods approach, the research encompasses an evaluation of the LLMs' general Geographic Information System (GIS) skills, general transportation domain knowledge as well as abilities to support human decision-making in the real-world transportation planning scenarios of congestion pricing. Results indicate that GPT-4 demonstrates superior accuracy and reliability across various GIS and transportation-specific tasks compared to Phi-3-mini, highlighting its potential as a robust tool for transportation planners. Nonetheless, Phi-3-mini exhibits competence in specific analytical scenarios, suggesting its utility in resource-constrained environments. The findings underscore the transformative potential of GenAI technologies in urban transportation planning. Future work could explore the application of newer LLMs and the impact of Retrieval-Augmented Generation (RAG) techniques, on a broader set of real-world transportation planning and operations challenges, to deepen the integration of advanced AI models in transportation management practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。