解决大模型生成时因无关上下文导致答案变差的问题
No-Worse Context-Aware Decoding: Preventing Neutral Regression in Context-Conditioned Generation
- 用双流结构在解码时判断上下文是否有效，无效则不依赖上下文
- 在无用上下文下准确率不降，有用上下文下仍能提升性能
- 适合需要可靠生成的问答和摘要任务
大语言模型在外部上下文（如检索到的证据）条件下可回答问题和总结文档，但上下文使用仍不可靠：即使上下文无信息量，模型也可能覆盖原本正确的输出（中性回归）。我们把中性回归形式化为‘不伤害’要求，并通过在一致答案上下文中对基准正确项的准确率下降来量化。提出一种解码时适配器NWCAD，基于双流架构和两阶段门控机制：当上下文无信息时退回到无上下文解码；否则在不确定性下采用类似CAD的回退策略进行上下文条件解码。在分离了‘不伤害可靠性’与‘上下文利用性’的基准上评估NWCAD，结果表明其在基准正确样本上防止了中性回归，同时在真正有帮助的上下文上保持了强准确性。
原文摘要 · Abstract (English)
Large language models (LLMs) can answer questions and summarize documents when conditioned on external contexts (e.g., retrieved evidence), yet context use remains unreliable: models may overwrite an already-correct output (neutral regression) even when the context is non-informative. We formalize neutral regression as a do-no-harm requirement and quantify it by measuring accuracy drops on baseline-correct items under answer-consistent contexts. We propose No-Worse Context-Aware Decoding (NWCAD), a decode-time adapter built on a two-stream setup with a two-stage gate: it backs off to no-context decoding when the context is non-informative, and otherwise uses context-conditioned decoding with a CAD-style fallback under uncertainty. We evaluate NWCAD on benchmarks that separate do-no-harm reliability from context utilization (accuracy gains on genuinely helpful contexts). NWCAD prevents neutral regression on baseline-correct items while preserving strong context-driven accuracy on helpful contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。