提出统一框架评估并优化复杂推理模型的质量与效率。
Characterizing, Evaluating, and Optimizing Complex Reasoning
- 用ME$^2$原则从宏观和微观层面定义推理质量。
- 将推理过程建模为有向无环图，实现对复杂结构的评估。
- 构建TRM-Preference数据集，训练思维奖励模型提升推理性能。
大型推理模型（LRMs）越来越依赖具有复杂内部结构的推理轨迹。然而，现有研究尚未统一回答三个基本问题：(1) 什么是高质量推理？(2) 如何可靠评估长且隐式结构的推理轨迹？(3) 如何利用评估信号优化推理过程？为此，本文提供统一视角：(1) 提出ME$^2$原则，从效率与有效性两个层面在宏观和微观上刻画推理质量；(2) 基于此原则，将推理轨迹建模为有向无环图（DAG），开发基于DAG的成对评估方法，捕捉复杂推理结构；(3) 基于此方法，构建TRM-Preference数据集，并训练思维奖励模型（TRM）实现大规模推理质量评估。实验表明，思维奖励可作为有效优化信号：测试时选择更优推理可带来最高19.3%的性能提升；强化学习训练中，思维奖励显著增强推理能力与整体表现，最高提升达3.9%，适用于多种任务。代码与数据见https://github.com/Simplified-Reasoning/TRM。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) increasingly rely on reasoning traces with complex internal structures. However, existing work lacks a unified answer to three fundamental questions: (1) what defines high-quality reasoning, (2) how to reliably evaluate long, implicitly structured reasoning traces, and (3) how to use such evaluation signals for reasoning optimization. To address these challenges, we provide a unified perspective. (1) We introduce the ME$^2$ principle to characterize reasoning quality along macro- and micro-level concerning efficiency and effectiveness. (2) Built on this principle, we model reasoning traces as directed acyclic graphs (DAGs) and develop a DAG-based pairwise evaluation method, capturing complex reasoning structures. (3) Based on this method, we construct the TRM-Preference dataset and train a Thinking Reward Model (TRM) to evaluate reasoning quality at scale. Experiments show that thinking rewards serve as an effective optimization signal. At test time, selecting better reasoning leads to better outcomes (up to 19.3\% gain), and during RL training, thinking rewards enhance reasoning and performance (up to 3.9\% gain) across diverse tasks. Code and data are available at https://github.com/Simplified-Reasoning/TRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。