让大模型学会在不懂时主动放弃回答，避免胡说八道。
KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning
- 用动态奖励机制实时判断模型知识边界，引导正确回答或合理拒绝
- 在多个测试集上同时提升准确率和减少幻觉，尤其在分布外数据表现更好
- 适合需要高可信度的问答系统，如医疗、法律等严谨场景
让大语言模型在知识范围之外适当放弃回答，是缓解幻觉的关键。现有强化学习方法虽能促进自主拒绝，但因静态奖励机制不考虑模型知识边界，常导致过度保守，牺牲准确率。本文提出KARL框架，通过持续对齐模型的拒绝行为与其演化的知识边界。核心创新包括：基于组内响应统计的在线知识边界估计的感知奖励机制，动态奖励正确回答或引导性拒绝；以及两阶段强化训练策略，先探索知识边界并避开‘拒绝陷阱’，再将边界外的错误回答转为拒绝，同时保持准确率。在多个基准测试上的实验表明，KARL在准确率与幻觉抑制之间取得更优平衡，有效降低幻觉，且在分布内与分布外场景均保持高准确率。
原文摘要 · Abstract (English)
Enabling large language models (LLMs) to appropriately abstain from answering questions beyond their knowledge is crucial for mitigating hallucinations. While existing reinforcement learning methods foster autonomous abstention, they often compromise answer accuracy because their static reward mechanisms, agnostic to models' knowledge boundaries, drive models toward excessive caution. In this work, we propose KARL, a novel framework that continuously aligns an LLM's abstention behavior with its evolving knowledge boundary. KARL introduces two core innovations: a Knowledge-Boundary-Aware Reward that performs online knowledge boundary estimation using within-group response statistics, dynamically rewarding correct answers or guided abstention; and a Two-Stage RL Training Strategy that first explores the knowledge boundary and bypasses the "abstention trap", and subsequently converts incorrect answers beyond the knowledge boundary into abstentions without sacrificing accuracy. Extensive experiments on multiple benchmarks demonstrate that KARL achieves a superior accuracy-hallucination trade-off, effectively suppressing hallucinations while maintaining high accuracy across both in-distribution and out-of-distribution scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。