用冷门编程语言测试大模型真实推理能力，发现性能暴跌。
EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
- 用5种冷门语言构建评测基准，模拟模型面对陌生语言的挑战。
- 同一问题在Python中准确率100%，在冷门语言中仅0-11%。
- 即使使用少样本和自我反思，也无法弥补能力差距，适合评估泛化能力。
大型语言模型在代码生成基准上表现接近顶尖水平，但这些基准（如SWE-bench和HumanEval）使用的编程语言（如Python、JavaScript）均在预训练语料中大量出现，并在后训练阶段被强化。为研究模型在不熟悉语言上的表现，我们引入EsoLang-Bench，采用五种图灵完备的冷门编程语言（Brainfuck、Befunge-98、Whitespace、Unlambda、Shakespeare）。这些语言虽与主流语言具备同等计算能力，但因设计复杂、预训练数据极少（比Python少340至6万余倍的GitHub仓库）、无实际部署价值，故对模型而言属于分布外场景。我们在五个前沿模型上测试五种提示策略，发现显著能力差距：相同80个算法问题在Python/JavaScript中准确率达100%，而在冷门语言中仅为0%-11%。少样本学习与自省机制亦无法弥合该差距。EsoLang-Bench因此提供了一个抗污染的测试平台，用于衡量前沿模型将算法求解能力泛化到训练分布外语言的能力。
原文摘要 · Abstract (English)
Large language models achieve near-ceiling performance on code generation benchmarks, yet most of the programming languages used by popular benchmarks such as SWE-bench and HumanEval (e.g. Python, JavaScript) are squarely in-distribution. They appear at scale in pre-training corpora and are heavily reinforced during post-training. To study LLM performance on unfamiliar programming languages, we introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare). All five of our chosen esoteric languages are Turing-complete, so the same algorithmic problems that are solvable in Python or JavaScript are in principle solvable in each of them. Yet, they are unfamiliar to LLMs which makes them a good proxy for evaluating out-of-distribution performance. The unfamiliarity of esoteric languages comprises of: (i) the hard-by-design primitives comprising the language; (ii) substantially less representation in pre-training corpora (340x to over 60,000x fewer public GitHub repositories than Python); (iii) negligible deployment value, which makes targeted inclusion in post-training data economically irrational. We evaluate five frontier models across five prompting strategies and find a dramatic capability gap. The same 80 problems expressed in Python or JavaScript reach 100% accuracy on top frontier models, while the equivalent esoteric versions score only 0-11%. Few-shot learning and self-reflection also fail to close this gap. EsoLang-Bench therefore provides a contamination-resistant testbed for measuring how well frontier models generalise algorithmic problem-solving to programming languages outside their training distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。