测试大模型在不同代码上的泛化能力，发现最新模型表现显著提升。
Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
- 构建四类代码数据：领域语言、LLM生成、编程竞赛和变异代码。
- 10个最新模型对比显示，新模型在跨类型代码上推理更鲁棒。
- 适合关注模型泛化能力的研究者和开发者参考。
我们评估大型语言模型（LLMs）在不同类型程序上的代码推理泛化能力。提出获取分布内与分布外代码的技术，包括领域特定语言代码、由LLM自动生成的代码、来自编程竞赛的数据，以及这些代码的变异版本。同时设计实验方法，通过比较模型在这些代码上的表现来评估泛化能力。对过去一年中10个最先进的模型进行广泛评估，揭示其随时间推移及在不同程序类别中的泛化特性。结果表明，早期模型表现出模式匹配行为，而最新模型在代码推理上展现出强大的泛化能力。
原文摘要 · Abstract (English)
We assess how the code reasoning abilities of large language models (LLMs) generalize to different kinds of programs. We present techniques for obtaining in- and out-of-distribution programs with different characteristics: code sampled from a domain-specific language, code automatically generated by an LLM, code collected from competitive programming contests, and mutated versions of these programs. We also present an experimental methodology for evaluating LLM generalization by comparing their performance on these programs. We perform an extensive evaluation across 10 state-of-the-art models from the past year, obtaining insights into their generalization capabilities over time and across different classes of programs. Our results highlight that while earlier models exhibit behavior consistent with pattern matching, the latest models exhibit strong generalization abilities on code reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。