让大模型互出难题,用对战评分判断推理能力。
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
- 模型自创编程谜题,互相挑战验证解法
- 10个前沿模型对战,排名与主流基准一致
- 无需人工出题,低成本测试推理与创造力
评估大语言模型的推理能力正变得愈发困难,因模型持续进步。以往依赖人类精心设计高难度问题(如博士级知识)的基准成本高昂,且存在训练数据泄露风险。本文受16世纪数学对决启发,提出「令牌对决」(The Token Games, TTG):模型通过自创编程谜题相互挑战。谜题格式为给定一个返回布尔值的函数,寻找使其返回真值的输入,支持灵活建模与自动验证。通过成对对决结果计算埃洛评分,实现模型间相对能力对比。在TTG上评估10个前沿模型,仅耗资不足200美元,无需人工构造题目,且排名与现有基准(如Humanity's Last Exam)高度一致。研究发现,当前模型仍难以生成优质谜题。该方法为推理评估提供了抗饱和新范式,可同时测试模型的创造力与任务生成能力。
原文摘要 · Abstract (English)
Evaluating the reasoning capabilities of Large Language Models is increasingly challenging as models improve. Human curation of hard questions is highly expensive, especially in recent benchmarks using PhD-level domain knowledge to challenge the most capable models. Even then, there is always a concern about whether these questions test genuine reasoning or if similar problems have been seen during training. Here, we take inspiration from 16th-century mathematical duels to design The Token Games (TTG): an evaluation framework where models challenge each other by creating their own puzzles. We leverage the format of Programming Puzzles - given a function that returns a boolean, find inputs that make it return True - to flexibly represent problems and enable verifying solutions. Using results from pairwise duels, we then compute Elo ratings, allowing us to compare models relative to each other. We evaluate 10 frontier models on TTG, and closely match the ranking from existing benchmarks such as Humanity's Last Exam, spending less than $200 USD and without involving any human effort in creating puzzles. We also find that creating good puzzles is still a highly challenging task for current models. Overall, our work suggests new paradigms for evaluating reasoning that avoid saturation by design, and that allow testing models for other skills like creativity and task creation alongside problem solving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。