用强化学习训练提示极化器,让大模型在误导信息下仍能选对答案。
LODESTAR: Robust Entropy-Based Answer Selection in Retrieval-Augmented Generation for Question Answering -- Directing Frozen-LLM Entropy with a Reinforcement-Learned Prompt Polarizer under Misleading Passages

- 用强化学习训练一个短提示极化器,干预冻结模型的输出熵。
- 在5个基准上提升F1至0.5339,超越所有已有方法。
- 适合需要高鲁棒性的问答系统研发者使用。
预测分布熵是检索增强生成(RAG)中强大的答案选择准则:在五个QA基准上,选择冻结语言模型生成的熵最低的答案,可使平均F1从0.4769提升至0.5148,无需黄金答案。然而该规则会失效——误导性段落使模型自信出错,导致熵降低,误判为可信。问题根源在于模型读取的上下文,而这是可干预的输入。我们提出LODESTAR:首个通过评估文本干预对第三方冻结模型不确定性影响来评分的方法。利用强化学习(GRPO)离线训练一个极化器——一段固定自然语言提示插入响应提示中,不修改模型权重,引导熵信号使其在误导信息下仍可靠。训练标签基于黄金答案与两名LLM裁判生成,推理时无需读取。在5,008个问题上,所有竞争方法使用相同冻结模型与候选池,LODESTAR达到最高平均F1(0.5339)、最高宏精确匹配(0.4136)及最高GPT-4o评分(0.6435)。其三种子均值在全部70个F1单元中胜出,且在F1上对14种已发布配置均显著领先。增益在NQ-Open(域内)和SQuAD、TriviaQA、EntityQuestions、WebQuestions(域外)均成立。消融实验显示,极化器使模型更少依赖误导段落(26.0% vs 30.3%)。
原文摘要 · Abstract (English)
Predictive-distribution entropy is a strong answer-selection rule in retrieval-augmented generation (RAG) for question answering: across five QA benchmarks, selecting the answer a frozen respondent LLM produces with the lowest answer-token entropy lifts mean $F_1$ from 0.4769 to 0.5148 over the retriever's top-ranked passage, without gold answers. Yet this rule, which prior entropy-based selectors adopt, fails: a misleading passage makes the respondent confidently wrong, driving entropy down where the uncertainty signal looks most trustworthy. The failure comes from the passage the respondent reads, and the context it is read in is an input we can intervene on. We introduce LODESTAR: to our knowledge the first method to score a text intervention by the uncertainty it induces in a third-party frozen respondent, compared within one question. LODESTAR uses reinforcement learning (GRPO) to train, once and offline, a polarizer -- a short fixed natural-language string inserted into the respondent's prompt and never into its weights, directing entropy so that entropy-based answer selection stays robust to misleading passages; training labels are built from gold answers and two LLM judges, and inference reads neither. With every competing selector under the same frozen respondent and candidate pools on 5,008 questions, LODESTAR attains the highest mean $F_1$ of any inference-ready selector (0.5339), the highest macro exact match (0.4136), and the highest GPT-4o judge score of the frozen-respondent configurations judged (0.6435); its three-seed mean wins all 70 $F_1$ cells against fourteen published configurations and is paired-significant on $F_1$ against every one. The gain holds in-domain on NQ-Open and out-of-domain over SQuAD, TriviaQA, EntityQuestions and WebQuestions. Ablating the polarizer shows it is what makes the respondent read a misleading passage less often (26.0% vs 30.3%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。