用强化学习让模型学会推理，精准定位大模型幻觉内容
Learning to Reason for Hallucination Span Detection
- 设计基于思维链的强化学习框架，鼓励模型逐步推理
- 在RAGTruth上超越基线模型，正确识别幻觉片段
- 适合需要高可信生成内容的研究与应用
大语言模型常产生无依据的幻觉内容，影响可靠性。现有方法多将幻觉检测视为二分类任务，但实际应用需定位具体幻觉段落，属多步决策过程。本文探究显式推理是否有助于该任务，通过对比带与不带思维链（CoT）的预训练模型，发现多次采样下思维链可提升正确答案概率。据此提出RL4HS强化学习框架，采用段落级奖励函数，结合组相对策略优化与类别感知策略优化，缓解奖励不平衡问题。在RAGTruth基准（摘要、问答、数据到文本）上的实验表明，该方法优于预训练推理模型与监督微调，证明了段落级奖励在幻觉段落检测中的必要性。
原文摘要 · Abstract (English)
Large language models (LLMs) often generate hallucinations -- unsupported content that undermines reliability. While most prior works frame hallucination detection as a binary task, many real-world applications require identifying hallucinated spans, which is a multi-step decision making process. This naturally raises the question of whether explicit reasoning can help the complex task of detecting hallucination spans. To answer this question, we first evaluate pretrained models with and without Chain-of-Thought (CoT) reasoning, and show that CoT reasoning has the potential to generate at least one correct answer when sampled multiple times. Motivated by this, we propose RL4HS, a reinforcement learning framework that incentivizes reasoning with a span-level reward function. RL4HS builds on Group Relative Policy Optimization and introduces Class-Aware Policy Optimization to mitigate reward imbalance issue. Experiments on the RAGTruth benchmark (summarization, question answering, data-to-text) show that RL4HS surpasses pretrained reasoning models and supervised fine-tuning, demonstrating the necessity of reinforcement learning with span-level rewards for detecting hallucination spans.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。