通过语义特征优化提问,主动减少大模型幻觉。
QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting
- 用17个语言特征构建奖励模型,动态优化提问重写策略。
- 在13个问答数据集上,胜过不重写基线87.5%。
- 揭示静态重写可能加剧幻觉,适合需精准控制的场景。
大型语言模型(LLMs)的高级推理能力导致幻觉现象增多;然而多数缓解方法仅关注事后过滤,而非塑造引发幻觉的查询。本文提出QueryBandits,一种基于强化学习的带状框架,通过最大化包含17种输入查询语言特征敏感度的幻觉倾向奖励模型,主动引导模型避免生成幻觉。在13个多样化QA基准上,每个数据集进行1,050次词汇扰动测试,采用汤普森采样(Thompson Sampling)的上下文最优策略相比无重写基线提升87.5%胜率,并分别优于零样本静态提示(“重述”或“扩展”)42.6%和60.3%。结果证实,通过查询重写干预可有效缓解幻觉。有趣的是,某些当前文献中常见的静态重写策略累积悔恨值高于无重写基线,表明其可能恶化幻觉。此外,各臂回归特征权重向量收敛显示,不存在对所有查询通用的最优重写策略。因此,借助语义特征的引导重写可通过前向传播机制显著改变输出行为,无需重新训练或梯度调整。
原文摘要 · Abstract (English)
Advanced reasoning capabilities in Large Language Models (LLMs) have caused higher hallucination prevalence; yet most mitigation work focuses on after-the-fact filtering rather than shaping the queries that trigger them. We introduce QueryBandits, a bandit framework that designs rewrite strategies to maximize a reward model, that encapsulates hallucination propensity based upon the sensitivities of 17 linguistic features of the input query-and therefore, proactively steer LLMs away from generating hallucinations. Across 13 diverse QA benchmarks and 1,050 lexically perturbed queries per dataset, our top contextual QueryBandit (Thompson Sampling) achieves an 87.5% win rate over a no-rewrite baseline and also outperforms zero-shot static prompting ("paraphrase" or "expand") by 42.6% and 60.3% respectively. Therefore, we empirically substantiate the effectiveness of QueryBandits in mitigating hallucination via the intervention that takes the form of a query rewrite. Interestingly, certain static prompting strategies, which constitute a considerable number of current query rewriting literature, have a higher cumulative regret than the no-rewrite baseline, signifying that static rewrites can worsen hallucination. Moreover, we discover that the converged per-arm regression feature weight vectors substantiate that there is no single rewrite strategy optimal for all queries. In this context, guided rewriting via exploiting semantic features with QueryBandits can induce significant shifts in output behavior through forward-pass mechanisms, bypassing the need for retraining or gradient-based adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。