用自演化生成器和检测器相互迭代，提升大模型幻觉识别能力。
Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator
- 检测器与生成器互为教练，通过强化学习不断进化。
- 小模型经训练后可超越大型模型的幻觉检测性能。
- 无需人工标注数据，适合研究幻觉检测与安全对齐的团队。
大语言模型输出中的忠实性幻觉识别仍因高质量标注数据稀缺而困难。现有方法依赖先进大模型合成训练数据（包括推理过程、标签和幻觉陈述），但将生成器视为静态组件，限制了检测器的持续优化。为此，我们提出幻觉自博弈（Hallucination Self-Play, HSP）框架，使检测器能借助演化的生成器进行自举训练。HSP 初始化两个角色：检测器评估输出忠实性，生成器则生成越来越难被检测的幻觉响应。首先，检测器在人工标注数据上微调，随后作为奖励模型，使用人工智能反馈的强化学习（RLAIF）训练生成器；接着，演进后的生成器生成幻觉数据，用于规则增强的强化学习进一步优化检测器。在RAGTruth基准及两种模型家族上的实验表明，该框架可使小型模型逐步提升至媲美甚至超越大型模型的幻觉检测性能，且无需外部监督。代码已公开。
原文摘要 · Abstract (English)
Identifying faithfulness hallucinations in LLM-generated outputs remains challenging due to the scarcity of high-quality annotated data. Recent work relies on advanced LLMs to synthesize training data, including rationales, labels, and hallucinated claims. However, these methods treat the generator as a static component, limiting iterative improvement of the detector. To address this limitation, we introduce Hallucination Self-Play (HSP), a novel framework that enables the detector to bootstrap with an evolved generator. HSP involves two roles initialized from the same base model, a detector that assesses the faithfulness of model outputs, and a generator that produces increasingly hard-to-detect hallucinated responses. Specifically, the detector is first fine-tuned on human-labeled data and then employed as a reward model to train the generator via reinforcement learning from AI feedback (RLAIF). In turn, the evolved generator synthesizes hallucination data to further optimize the detector through rule-based reinforcement learning. Experiments on RAGTruth benchmark and two model families demonstrate that the proposed framework can progressively enhance a small LLM to match or even outperform advanced LLMs without external supervision. Our code is available at https://anonymous.4open.science/r/Hallucination-Self-Play-50B5 .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。