arXiv:2608.06377cs.CLcs.AI2026-08被引 1

让大模型学会在该信时信,不该信时不盲目跟从。

Learning When to Trust via Selective Context Preference Optimization

论文配图:Learning When to Trust via Selective Context Preference Optimization
图 1 · 摘自论文原文
  • 设计新方法SCOPE,通过对比学习优化信任判断
  • 在多个模型上显著降低错误翻转率(SC2W)
  • 适合需要可靠推理的场景如医疗、法律

语言模型越来越依赖外部信息,但一个误导性信号就可能使正确答案变错。单纯训练模型忽略所有上下文看似稳健,实则无用。本文提出‘选择性信任’的新视角,构建MIST基准,涵盖四种匹配情境(纯净、误导、正确上下文、无关上下文),并引入SC2W指标,衡量误导信号导致正确答案出错的频率。在广泛测试中发现此类脆弱性普遍存在。为此提出SCOPE方法,基于清洁-正确与误导-错误的失败对,对标准直接偏好优化(DPO)目标进行平衡优化,而非仅针对误导项。实验显示,该方法在主流开源模型上大幅降低SC2W值,同时保持在干净、正确或无关上下文下的准确率。本文主张以选择性信任能力作为模型评估核心标准。

原文摘要 · Abstract (English)

Language models increasingly condition their answers on external signals, and a single misleading one can turn a correct answer wrong. The obvious remedy, training models to resist such signals, hides a failure mode: a model that ignores all context looks robust yet is useless when the context is worth trusting. We recast the problem as selective trust and introduce MIST, a human-annotated benchmark that renders each reasoning item under four matched conditions (clean, misleading, correct-context, and irrelevant-context), together with SC2W, a paired metric counting how often a misleading signal flips a clean-correct answer to wrong. Across a comprehensive benchmark study, we observe that such a susceptibility is universal. We then propose SCOPE, which mines clean-correct/misleading-wrong failures and optimizes a standard Direct Preference Optimization (DPO) objective over matched preference pairs balanced equally across all four conditions, rather than over misleading items alone. Our approach substantially reduces SC2W on popular open-sourced models while preserving accuracy when the added context is clean, correct, or irrelevant. With this work, we argue that models should be judged on selective trust, not on resistance alone.

模型可信度偏好优化推理安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。