测试大模型是否过度思考，发现很多模型会无意义地多生成70倍内容。
DNR Bench: Benchmarking Over-Reasoning in Reasoning LLMs
- 设计150个陷阱式问题，检验模型能否识别无需推理的场景
- 顶尖推理模型生成内容最多达必要量的70倍，准确率反而更低
- 适合关注模型效率与实用性的研究者和工程师参考
推理增强型大语言模型在测试时扩展（test-time scaling）的推动下性能显著提升，能够进行更深层次的推理以解决复杂问题。然而，这种增强也带来了冗余的令牌生成和不必要的求解尝试。本文提出Dont Reason Bench（DNR Bench），一个新基准，用于评估大模型在理解复杂推理触发条件时的鲁棒性，以及避免无效生成的能力。DNR Bench包含150个经过对抗性设计的提示，对人类而言容易理解并能快速响应，但对许多近期主流的推理大模型却难以应对。该基准测试模型在指令遵循、幻觉规避、冗余过滤和无法回答问题识别等多个能力维度的表现。我们评估了包括DeepSeek-R1、OpenAI O3-mini、Claude-3.7-sonnet在内的推理大模型（RLMs），并与强大的非推理模型GPT-4o进行对比。实验结果表明，推理模型生成的令牌数量最多可达实际所需量的70倍，且在某些任务上表现远不如非推理模型，准确率更低。这些发现凸显了在推理大模型中亟需更高效的训练与推理策略。
原文摘要 · Abstract (English)
Test-time scaling has significantly improved large language model performance, enabling deeper reasoning to solve complex problems. However, this increased reasoning capability also leads to excessive token generation and unnecessary problem-solving attempts. We introduce Dont Reason Bench (DNR Bench), a new benchmark designed to evaluate LLMs ability to robustly understand the tricky reasoning triggers and avoiding unnecessary generation. DNR Bench consists of 150 adversarially designed prompts that are easy for humans to understand and respond to, but surprisingly not for many of the recent prominent LLMs. DNR Bench tests models abilities across different capabilities, such as instruction adherence, hallucination avoidance, redundancy filtering, and unanswerable question recognition. We evaluate reasoning LLMs (RLMs), including DeepSeek-R1, OpenAI O3-mini, Claude-3.7-sonnet and compare them against a powerful non-reasoning model, e.g., GPT-4o. Our experiments reveal that RLMs generate up to 70x more tokens than necessary, often failing at tasks that simpler non-reasoning models handle efficiently with higher accuracy. Our findings underscore the need for more effective training and inference strategies in RLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。