测试大模型逐步推理的准确性,发现越长步骤错误越多。
L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution
- 用合成代码生成逐步执行轨迹,评估模型推理过程正确性
- 步骤越多错误率越高,大模型和增强模型表现更好
- 适合研究推理能力提升与模型可靠性改进的团队
复杂推理任务常依赖在多步中稳定准确地应用简单规则,我们称之为“零级”推理。为系统评估此能力,提出 L0-Bench 基准,用于评测程序正确性——即生成无错误推理过程的能力,补充现有主要关注结果正确性的基准。L0-Bench 给出带简单操作的合成 Python 函数,评分模型生成的分步执行轨迹是否正确。其合成特性支持沿多个维度(如轨迹步骤数)可扩展生成测试程序。评估多种近期闭源与开源模型在基线测试集上的表现,所有模型在目标轨迹步骤增加时均出现性能下降,而更大模型及推理增强模型能更好地保持多步正确性。此外,利用 L0-Bench 探索测试时缩放:输入上下文长度、多数投票解的数量、推理步数。结果表明‘零级’推理仍有较大提升空间,且指明构建更可靠推理系统的潜在方向。
原文摘要 · Abstract (English)
Complex reasoning tasks often rely on the ability to consistently and accurately apply simple rules across incremental steps, a foundational capability which we term "level-0" reasoning. To systematically evaluate this capability, we introduce L0-Bench, a language model benchmark for testing procedural correctness -- the ability to generate correct reasoning processes, complementing existing benchmarks that primarily focus on outcome correctness. Given synthetic Python functions with simple operations, L0-Bench grades models on their ability to generate step-by-step, error-free execution traces. The synthetic nature of L0-Bench enables systematic and scalable generation of test programs along various axes (e.g., number of trace steps). We evaluate a diverse array of recent closed-source and open-weight models on a baseline test set. All models exhibit degradation as the number of target trace steps increases, while larger models and reasoning-enhanced models better maintain correctness over multiple steps. Additionally, we use L0-Bench to explore test-time scaling along three dimensions: input context length, number of solutions for majority voting, and inference steps. Our results suggest substantial room to improve "level-0" reasoning and potential directions to build more reliable reasoning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。