让大模型先反思再回答,减少幻觉同时保持高准确率。
REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignment

- 引入思考-反思-作答结构,强制模型自我检查推理过程。
- 通过鼓励不确定时直接说‘不知道’,使幻觉率降低58%~72%。
- 仅需一次前向传播,适配多种模型且无需额外监督或搜索。
大型推理模型容易产生幻觉,严重影响其可靠性并阻碍安全部署。幻觉主要来自两类原因:推理幻觉(错误推导导致错误结论)和知识幻觉(缺乏必要事实知识)。为应对推理幻觉,本文提出REIN框架,训练模型生成结构化推理序列:“<think>” → “<reflection>” → “<answer>”,实现回答前的显式自我反思。为应对知识幻觉,REIN设计奖励机制,当所有采样的推理链均无法得出正确答案时,鼓励模型明确拒绝回答(如“我不知道”),避免无依据预测。在数学与常识推理基准上的大量实验表明,REIN持续提升选择性准确率,显著减少错误但自我认可的回答,且保持高覆盖率。值得注意的是,该方法仅需单次前向传播,无需过程监督、推理时控制器、外部搜索或多轮批判。多骨干模型实验显示,与基线相比,REIN使幻觉代理指标降低58%~72%,平均覆盖率维持在86%~91%,对尝试回答的问题选择性准确率提升6.6%~14.2%。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) are prone to hallucination, which undermines their reliability and poses challenges for safe deployment. Hallucinations in LRMs arise from two distinct failure sources: reasoning hallucination, where flawed inference steps propagate to an incorrect conclusion, and knowledge hallucination, where the model lacks the requisite factual knowledge to answer the query. To address reasoning hallucination, we propose REIN, an alignment framework that trains LRMs to produce a structured reasoning sequence, $\texttt{<think>} $$\rightarrow$ $\texttt{<reflection>} $$\rightarrow$ $\texttt{<answer>}$, enabling explicit self-reflection before committing to a final answer. To address knowledge hallucination, REIN introduces a reward mechanism that encourages explicit abstention (e.g., "I don't know") when none of the sampled reasoning chains yields a correct answer, allowing the model to refrain from unsupported predictions. Extensive evaluations on mathematical and commonsense reasoning benchmarks show that REIN consistently improves selective accuracy, reduces incorrect-but-self-endorsed responses, and maintains high coverage compared with competitive baselines. Notably, REIN achieves these gains within a single forward pass, without requiring process supervision, inference-time controllers, external search, or multi-round critiques. Experiments on multiple backbones show that REIN reduces the hallucination proxy by $58\sim72\%$ relative to the base models while maintaining $86\sim91\%$ average coverage, and improves selective accuracy on attempted questions by $6.6\sim14.2\%$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。