测试大模型在组合数学代码生成中的表现，发现零样本更优且推理耗尽输出预算会失败。
FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
- 基于FindStat构建2329个任务的执行基准，仅允许生成代码函数
- 统计合成比映射合成容易，五例提示反而降低经典双射的准确率
- 长提示导致精度骤降，适合评估代码生成与符号归纳能力的模型
我们提出FindStatBench，一个用于评估大语言模型在组合代码合成任务上的执行基准。该基准源自FindStat，包含2,329个任务，覆盖24个类别和552万条隐藏实例，涵盖统计合成（将对象映射为整数）与映射合成（将对象映射为对象）。每个任务提供数学描述及最多五个公开输入-输出示例；模型需生成一个无检索、无工具、无执行反馈、无投票或重排的Python求解函数。提交结果通过在保留的组合对象上沙盒化精确执行评分。我们评估了十一个系统：四个闭源生产模型和七个通过单一推理服务提供的开源模型。结果显示：最强的开闭源系统在实例准确率上趋于一致，对所有系统取最优或五路采样仅带来有限提升；示例可能有害，部分经典双射在零示例下完美解决，但在五示例提示下失败；部分失败源于输出预算机制，推理过程可能在代码生成前耗尽可见响应。总体而言，统计合成远比映射合成简单，某些集合准确率仍接近零，长提示引发显著精度下降，精确符号规则归纳依然脆弱。
原文摘要 · Abstract (English)
We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, it contains 2,329 tasks across 24 collections and 5.52M hidden instances, covering statistic synthesis, which maps objects to integers, and map synthesis, which maps objects to objects. Each task gives a mathematical description and at most five public input-output examples; a model must emit one Python solve function with no retrieval, tool use, execution feedback, voting, or reranking. Submissions are scored by exact sandboxed execution on held-out combinatorial objects. We evaluate eleven systems: four closed-source production models and seven open-weight models served through one inference provider. FindStatBench reveals three main patterns. First, the strongest open- and closed-source systems converge within 1 pp instance accuracy, and both an oracle over all systems and five-way sampling from one mid-tier model yield only limited task-accuracy gains. Second, examples can hurt: several classical bijections are solved perfectly with zero examples but fail under five-example prompts. Third, some failures reflect output-budget mechanics, as reasoning can exhaust the visible response before code is emitted. Overall, statistic synthesis is much easier than map synthesis, some collections remain near-zero, long prompts cause a sharp accuracy cliff, and exact symbolic rule induction remains brittle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。