让大模型学会从混杂信息中挑有用内容,拒有害误导。
Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results
- 用规则或语义判断器指导模型筛选证据,避免全拒或全信。
- 筛选准确率从22.46%提升至26.46%,有害内容采纳减少。
- 保留通用能力,适合需高可靠性检索的场景使用。
检索增强的大语言模型常面临有效信息与误导性内容混杂的问题。盲目拒绝会丢弃有效信息,不加甄别则产生错误或不安全回答。因此,能否在复杂检索结果中选择性采纳相关证据、拒绝欺骗性内容,对实际部署至关重要。本文提出SelectBench,一个用于选择性证据采纳的可控基准与训练集,并基于DAPO方法对Qwen3.5-4B进行后训练,采用确定性规则奖励或冻结的语义判别器。在修正后的325个样本的SelectBench-v2测试集上,严格成功率达22.46%(原始检查点)提升至25.54%(DAPO-Rule)和26.46%(DAPO-DeepSeek)。两种训练策略均降低了违规内容采纳率,生成更短且聚焦的回答,但提示注入攻击未带来提升。成对增益虽显著但未通过霍尔姆校正,表明需更强的奖励设计或更多训练迭代以获得稳健改进。DAPO-DeepSeek在MMLU和干净的HotpotQA上无明显性能下降,说明后训练保留了通用能力。结果表明选择性证据利用方向性改善,但抗注入与统计鲁棒性仍是未来关键挑战。
原文摘要 · Abstract (English)
Retrieval-augmented large language models frequently face contexts that interleave useful evidence with misleading statements or instruction-like content. Blanket refusal discards valid evidence, whereas uncritical adoption yields incorrect or unsafe answers. The ability to selectively adopt relevant information while rejecting deceptive or harmful content is therefore critical for reliable deployment in real-world retrieval settings. We introduce SelectBench, a controlled benchmark and training set for selective evidence adoption, and post-train Qwen3.5-4B directly with DAPO using either deterministic rule rewards or a frozen semantic judge. On the corrected 325-example SelectBench-v2 test set, strict success rises from 22.46% for the original checkpoint to 25.54% with DAPO-Rule and 26.46% with DAPO-DeepSeek. Both trained policies reduce forbidden-content adoption and produce shorter, more focused responses, yet prompt-injection following does not improve. The paired gains are modest and fail to survive Holm correction, suggesting that stronger reward shaping or additional training iterations may be needed for more robust gains. DAPO-DeepSeek exhibits no material degradation on MMLU or clean HotpotQA, indicating that the post-training procedure preserves general capabilities. These results demonstrate a directional improvement in selective evidence use, while identifying injection resistance and statistical robustness as important remaining challenges for future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。