用广播谜题测试大模型常识推理能力，让普通人也能看懂模型表现。
ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
- 基于美国广播节目谜题构建613道常识推理题
- o1模型在该基准上显著领先其他模型
- 揭示模型在思考中断、自认错误等新缺陷
现有前沿模型评测多聚焦于专业级知识，非专业人士难以理解。本文提出一个基于美国国家公共电台（NPR）周日谜题挑战的基准，包含613个问题，仅需一般常识即可理解。该基准对人类和模型均具挑战性，但正确答案易于验证，模型错误也容易识别。随着大语言模型广泛部署，我们主张开发人类可理解、无需专业知识的评测体系。实验发现：OpenAI o1在本基准上显著优于其他推理模型，尽管在专业知识测试中与其他模型持平。分析其推理过程还揭示了新型失败模式：DeepSeek R1常在给出错误答案前主动放弃（‘I give up’），表现出高度不确定性，极少数情况下甚至未完成思考即超出上下文窗口限制。我们还量化了延长推理的边际收益，明确了进一步思考不再提升准确性的临界点。
原文摘要 · Abstract (English)
Existing benchmarks for frontier models often test specialized, "PhD-level" knowledge that is difficult for non-experts to grasp. In contrast, we present a benchmark with 613 problems based on the NPR Sunday Puzzle Challenge that requires only general knowledge. Our benchmark is challenging for both humans and models; however correct solutions are easy to verify, and models' mistakes are easy to spot. As LLMs are more widely deployed in society, we believe it is useful to develop benchmarks for frontier models that humans can understand without the need for deep domain expertise. Our work reveals capability gaps that are not evident in existing benchmarks: OpenAI o1 significantly outperforms other reasoning models on our benchmark, despite being on par with other models when tested on benchmarks that test specialized knowledge. Furthermore, our analysis of reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance, often concedes with "I give up" before providing an answer that it knows is wrong. R1 can also be remarkably "uncertain" in its output and in rare cases, it does not "finish thinking," which suggests the need for techniques to ``wrap up'' before the context window limit is reached. We also quantify the effectiveness of reasoning longer to identify the point beyond which more reasoning is unlikely to improve accuracy on our benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。