构建首个可执行的积木编程自然语言生成评测基准
NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming
- 提出可执行的积木编程NL2Code评测集，包含31万对真实项目数据
- 引入语义一致性度量SAC，发现93%字面匹配模型仍存在语义错位
- 揭示长任务中动作、条件等关键槽位错误，适合教育类AI研究者
积木式编程环境如Scratch广泛应用于早期编程教学，但自然语言到代码（NL2Code）研究主要聚焦于文本语言。Scratch程序具有事件驱动、视觉组合和并发脚本分布等特点，传统NL2Code假设与评估方法难以适用。本文提出NL2Scratch，一个包含311,648个解析有效自然语言-程序对的可执行基准，其程序源自真实Scratch项目，并与语义对齐的自然语言描述配对。为实现可靠评估，提出可解释的槽位级语义一致性度量SAC，构建23,594个语义验证样本池及800个槽位平衡诊断集。在指令微调与精调大模型上的实验表明，词级F1超过0.93的模型往往无法达到完美SAC，尤其在长任务中，操作槽位如动作、条件和数值参数错误集中，暴露了传统指标下不可见的失败模式。
原文摘要 · Abstract (English)
Block-based programming environments such as Scratch are widely used in early programming education, yet natural-language-to-code (NL2Code) research has focused primarily on text-based languages. Scratch programs are event-driven, visually compositional, and distributed across concurrent scripts, making conventional NL2Code assumptions and evaluation insufficient. We introduce NL2Scratch, an executable benchmark for natural-language-to-Scratch generation comprising 311,648 parser-valid NL--program pairs, whose program side is extracted from real Scratch projects and paired with semantically aligned NL descriptions. For reliable evaluation beyond surface overlap, we propose Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement between descriptions and programs. With SAC, we construct a semantically validated pool of 23,594 examples, and a slot-balanced 800 diagnostic benchmark. Experiments across instruction-tuned and fine-tuned LLMs reveal a notable gap between lexical similarity and semantic alignment: models achieving token-level F1 above 0.93 often fail to attain perfect SAC, particularly on longer examples. Errors concentrate on operational slots like actions, conditions, and numeric arguments, exposing failure modes largely invisible under conventional metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。