arXiv:2607.19678cs.CLcs.AI2026-07

无需参考答案,通过推理链分析评估大模型问答质量

Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

论文配图:Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
图 1 · 摘自论文原文
  • 将生成的推理过程拆解为片段,用自然语言推理识别段落间逻辑关系
  • 构建超图结构并用确定性搜索标注每个段落的可信度,准确率超基准模型15%
  • 特别适合医疗、数学等需严谨推理的高风险场景,对误判有强敏感性

高风险领域中,大模型生成的答案常流于表面但难以验证,尤其当涉及多步推理时。本文提出一种基于推理的无参考评估框架,将生成的推理链分解为片段,利用自然语言推理(NLI)标注局部前提-目标关系,并组织成超图结构。采用确定性逆向的与-或搜索,为每个片段分配审计标签,反映其在回答中的支撑程度。在两个场景下评估:使用Hard2Verify进行演绎数学推理,以及使用新构建的医生标注数据集UroReason(来自真实临床案例的LLM推理链)。结果表明,该方法比直接用大模型作为评判者更可靠。在临床场景中,当前先进模型常无法识别问题推理段,过度接受看似流畅但根基薄弱的回答。研究显示,问答评估必须考虑推理链中推断关系的组合方式,不能仅依赖最终答案或大模型自评。UroReason将通过API提供,代码也将开源。

原文摘要 · Abstract (English)

AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer. We propose a reasoning-based, reference-free framework for auditing LLM-generated outputs. The method decomposes a generated reasoning trace into segments, labels local premise-target relations using Natural Language Inference (NLI), and organizes these relations into a hypergraph. A deterministic backward AND-OR search then assigns segment-level audit labels that indicate how each segment is grounded within the generated response. We evaluate the framework in two settings: deductive mathematical reasoning with Hard2Verify, and open-ended medical reasoning with UroReason, a new physician-annotated benchmark of LLM reasoning traces from real clinical cases. Across these settings, our NLI-hypergraph audit provides a more reliable reference-free evaluation signal than direct LLM-as-judge baselines. In the clinical setting, state-of-the-art LLM judges often fail to identify problematic reasoning segments, over-accepting fluent but weakly grounded responses. Our results show that QA evaluation should account for how inferential relations compose across a reasoning trace, rather than relying only on final answers or LLMs as verifiers. UroReason will be made available through an API, and our code will be released as open source.

推理评估无参考医疗AI大模型审计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。