对比大模型在汽车场景生成中的代码生成能力，发现中等规模开源模型性价比高。
David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation
- 构建NL2Scenic数据集与评估框架，含30个难度分层测试用例
- 编辑相似度(EDIT-SIM)最贴近人工评价，提出新指标EDIT-COMP提升评估可靠性
- 140亿参数开源模型性能达闭源顶级模型88%，适合本地部署
场景仿真对自动驾驶系统测试至关重要。针对CARLA平台的领域特定语言Scenic，现有大模型实现自然语言到Scenic代码生成面临数据稀缺、可复现性差和评估不一致问题。本文提出NL2Scenic——一个包含146组自然语言/Scenic代码对的开源数据集与评估框架，包含30个分层难度测试用例、示例检索器及14种提示策略（ZS, FS, CoT, SP, MoT）。评估了13个模型：4个闭源（GPT-4o, GPT-5, Claude-Sonnet-4, Gemini-2.5-pro）与9个开源代码模型（Qwen2.5Coder 0.5B–32B；CodeLlama 7B/13B/34B），使用文本指标（BLEU, ChrF, EDIT-SIM, CrystalBLEU）与执行指标（编译通过率、生成成功率），并结合专家评测（n=11）。结果表明，EDIT-SIM与人工判断相关性最强，提出新指标EDIT-COMP（EDIT-SIM与编译成功率的F1值），显著提升整体排名准确性。GPT-4o表现最佳，而Qwen2.5Coder-14B在本地硬件上达到其专家评分的约88%。基于示例检索的少样本提示（FSER）持续提升小模型性能，模型规模增长至中等后收益递减，且同规模下Qwen2.5Coder优于CodeLlama。NL2Scenic与EDIT-COMP为场景生成提供了标准化、可复现的评估基础，表明中等规模开源模型是自动驾驶场景编程的实际高效选择。
原文摘要 · Abstract (English)
Scenario simulation is central to testing autonomous driving systems. Scenic, a domain-specific language (DSL) for CARLA, enables precise and reproducible scenarios, but NL-to-Scenic generation with large language models (LLMs) suffers from scarce data, limited reproducibility, and inconsistent metrics. We introduce NL2Scenic, an open dataset and framework with 146 NL/Scenic pairs, a difficulty-stratified 30-case test split, an Example Retriever, and 14 prompting variants (ZS, FS, CoT, SP, MoT). We evaluate 13 models: four proprietary (GPT-4o, GPT-5, Claude-Sonnet-4, Gemini-2.5-pro) and nine open-source code models (Qwen2.5Coder 0.5B-32B; CodeLlama 7B/13B/34B), using text metrics (BLEU, ChrF, EDIT-SIM, CrystalBLEU) and execution metrics (compilation and generation), and compare them with an expert study (n=11). EDIT-SIM correlates best with human judgments; we also propose EDIT-COMP (F1 of EDIT-SIM and compilation) as a robust dataset-level proxy that improves ranking fidelity. GPT-4o performs best overall, while Qwen2.5Coder-14B reaches about 88 percent of its expert score on local hardware. Retrieval-augmented prompting, Few-Shot with Example Retriever (FSER), consistently boosts smaller models, and scaling shows diminishing returns beyond mid-size, with Qwen2.5Coder outperforming CodeLlama at comparable scales. NL2Scenic and EDIT-COMP offer a standardized, reproducible basis for evaluating Scenic code generation and indicate that mid-size open-source models are practical, cost-effective options for autonomous-driving scenario programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。