用论证结构评估大模型推理过程，比只看答案更准。
TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
- 基于图灵论证理论，分析推理链条的逻辑结构。
- 在7个模型上与基准准确率相关性达0.74。
- 可作为强化学习奖励信号，提升模型表现。
评估大语言模型（LLMs）的开放性输出仍具挑战性，因缺乏真实答案。现有指标依赖最终答案正确性或表面统计特征，忽视推理过程本身。我们提出TRACE（Toulmin-based Reasoning Assessment through Constructive Elements），一种分析思维链（CoT）推理过程的度量方法。不同于仅判断结果，TRACE通过融合图灵论证理论与弗拉维尔元认知框架，评估论证结构的合理性。在7个推理模型上对26.3K个问答样本的实验表明，其与基准准确率的相关系数达r=0.74。此外，TRACE作为强化学习奖励信号时，优于仅基于准确率的基线模型。结果表明，逻辑严谨的推理能产生更高质量的答案。因此，TRACE可作为开放性输出评估的互补工具。代码已开源：https://github.com/hyyangkisti/trace。
原文摘要 · Abstract (English)
Evaluating open-ended outputs from large language models (LLMs) remains challenging due to the absence of ground truth. Existing metrics rely on final-answer accuracy or surface-level statistics, leaving the reasoning process itself unexamined. We introduce TRACE (Toulmin-based Reasoning Assessment through Constructive Elements), a metric that analyzes Chain-of-Thought (CoT) reasoning processes. Rather than judging outcomes, TRACE inspects how arguments are constructed by integrating Toulmin's argumentation theory with Flavell's metacognitive framework to assess reasoning structure. Experiments on 26.3K QA samples across 7 reasoning models show strong correlation with benchmark accuracy (r=0.74). Furthermore, TRACE is effective as a reinforcement learning reward signal, outperforming accuracy-only baselines. Together, these results indicate that logically sound reasoning leads to higher-quality answers. TRACE thus serves as a complementary metric for evaluating open-ended outputs. Code is available at https://github.com/hyyangkisti/trace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。