用动态生成算法题评估大模型推理能力，避免数据泄露干扰
BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- 通过实时生成数学问题，确保测试不被训练数据污染
- 在高难度任务中，顶级模型准确率最高仅56.21%
- 适合关注模型真实推理能力的研究者与开发者
评估语言模型的公平性日益困难，因静态基准可能被训练数据污染，掩盖模型是真正推理还是记忆。我们提出BeyondBench，一种基于算法题动态生成的评估框架，确保每个测试实例均未被污染。该框架涵盖44个算法任务，共117种变体，分为三个难度等级：简易套件（29项）用于算术与统计，中等套件（5项，49种变体）用于序列模式与推理，困难套件（10项，68种变体）用于NP完全与约束满足问题。每项任务源自超过10^15种唯一实例，解决方案可确定性验证。我们评估了101个语言模型（85个开源，16个闭源），参数量从0.5B到141B，覆盖多种量化方案，采用三重评估以增强鲁棒性。结果显示，模型推理能力随复杂度显著下降，在困难套件中，Gemini-2.5-pro、Llama-3.3-70B和Qwen2.5-72B准确率分别为56.21%、27.16%和33.37%。无工具使用时性能大幅下滑，GPT-5、GPT-5-mini和GPT-5-nano整体准确率分别下降16.81%、15.86%和43.95%。污染抵抗依赖三大保证：(i) 问题空间远超任何静态数据集，(ii) 每个实例有可确定性验证解，(iii) 同构变换生成语义等价但语法新颖的问题。BeyondBench通过真正的算法求解重新定义推理评估。排行榜见https://ctrl-gaurav.github.io/BeyondBench/，Python包为beyondbench，代码库在https://github.com/ctrl-gaurav/BeyondBench。
原文摘要 · Abstract (English)
Evaluating language models fairly is increasingly difficult as static benchmarks risk contamination by training data, obscuring whether models truly reason or recall. We introduce BeyondBench, an evaluation framework using algorithmic problem generation to create mathematically grounded problems on the fly, ensuring each test remains uncontaminated. Our framework covers 44 algorithmic tasks with 117 variations across three difficulty levels: the Easy Suite (29 tasks) for arithmetic and statistics, the Medium Suite (5 tasks, 49 variations) for sequence patterns and reasoning, and the Hard Suite (10 tasks, 68 variations) for NP-complete and constraint satisfaction problems. Each task draws from a space exceeding 10^15 unique instances, with deterministically verified solutions. We evaluated 101 language models (85 open-source, 16 closed-source), spanning 0.5B to 141B parameters and multiple quantization schemes, using three-fold evaluation for robustness. Results reveal consistent reasoning deficiencies, with performance degrading sharply as complexity increases. In Hard Suite evaluations, Gemini-2.5-pro, Llama-3.3-70B, and Qwen2.5-72B achieved accuracies of 56.21%, 27.16%, and 33.37% respectively. Performance drops significantly without tool usage, with GPT-5, GPT-5-mini, and GPT-5-nano showing declines of 16.81%, 15.86%, and 43.95% in overall accuracy. Contamination resistance rests on three guarantees: (i) the problem space vastly exceeds any static dataset, (ii) every instance has a deterministically verifiable solution, and (iii) isomorphic transformations yield semantically equivalent but syntactically novel problems. BeyondBench redefines reasoning evaluation via genuine algorithmic problem-solving. Our leaderboard is at https://ctrl-gaurav.github.io/BeyondBench/, Python package at https://pypi.org/project/beyondbench/, and codebase at https://github.com/ctrl-gaurav/BeyondBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。