arXiv:2507.10541cs.CL2025-07ACL被引 15

用多题并发测试大模型,发现其真实推理能力远低于单题表现。

REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once

  • 同时出多道题,模拟真实场景下的认知压力
  • 顶尖模型在多任务下准确率暴跌,暴露隐藏缺陷
  • 适合评估模型抗干扰和资源分配能力,非纯解题

近期大型推理模型(LRMs)在特定任务基准上取得显著进展,但评估方法仍受限于孤立的单题求解范式。现有基准主要通过顺序测试单个问题,存在两大局限:(1) 易受数据污染,挑战性不足(如 DeepSeek-R1 在 MATH500 上达 97.0%),需大量人力生成新题;(2) 无法评估模型在多上下文压力下的表现,而这是实际部署的关键需求。为此,我们提出 REST(Reasoning Evaluation through Simultaneous Testing),一种通过同时呈现多个问题来对 LRMs 进行压力测试的框架。除基本推理外,REST 还评估模型在上下文优先级分配、跨问题干扰抵抗和动态认知负荷管理等方面的性能。评估发现:即使最先进的模型如 DeepSeek-R1,在压力测试中也出现显著性能下降。关键的是,REST 比现有基准具有更强区分能力,能揭示在单题测试中表现相近的模型之间的明显差异。分析显示:(1) “过度思考陷阱”是导致性能下降的关键因素;(2) 经过“long2short”训练的模型在多重压力下更能保持单题精度,优于标准训练模型。这些结果表明,REST 是一种成本更低、更具未来适应性的评估范式,更贴近真实推理需求,同时减少对持续人工标注的依赖。代码与结果见 https://opendatalab.github.io/REST。

原文摘要 · Abstract (English)

Recent Large Reasoning Models (LRMs) have achieved remarkable progress on task-specific benchmarks, yet their evaluation methods remain constrained by isolated problem-solving paradigms. Existing benchmarks predominantly assess single-question reasoning through sequential testing, resulting critical limitations: (1) vulnerability to data contamination and less challenging (e.g., DeepSeek-R1 achieves 97.0% on MATH500), forcing costly creation of new questions with large human efforts, (2) failure to evaluate models under multi-context pressure, a key requirement for real-world deployment. To bridge this gap, we present REST (Reasoning Evaluation through Simultaneous Testing), a stress-testing framework that exposes LRMs to multiple problems simultaneously. Beyond basic reasoning, REST evaluates several under-tested capabilities: contextual priority allocation, cross-problem interference resistance, and dynamic cognitive load management. Our evaluation reveals several striking findings: Even state-of-the-art (SOTA) models like DeepSeek-R1 exhibit substantial performance degradation under stress testing. Crucially, REST demonstrates stronger discriminative power than existing benchmarks, revealing pronounced performance differences among models that exhibit similar, near-ceiling performance under single-question evaluations. Some key insights emerge from our analysis: (1) the "overthinking trap" is a critical factor contributing to the performance degradation; (2) the models trained with "long2short" technique preserve more accuracy of their single-problem performance under REST, outperforming standard-trained counterparts. These results establish REST as a cost-efficient, future-proof evaluation paradigm that better reflects real-world reasoning demands while reducing reliance on continuous human annotation. Code and results are available at https://opendatalab.github.io/REST.

大模型评估多任务测试推理能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。