arXiv:2604.23750cs.LGcs.AI2026-04被引 2

解决大模型快速适配中知识冲突导致的性能下降问题

The Override Gap: A Magnitude Account of Knowledge Conflict Failure in Hypernetwork-Based Instant LLM Adaptation

论文配图:The Override Gap: A Magnitude Account of Knowledge Conflict Failure in Hypernetwork-Based Instant LLM Adaptation
图 1 · 摘自论文原文
  • 发现冲突失败源于适配器幅度恒定,而预训练知识强度随频率增长
  • 提出幅度增强策略,使深层冲突准确率提升至71%以上
  • 无需训练即可改善冲突处理,适合需要快速知识更新的场景

基于超网络的方法如Doc-to-LoRA可在单次前向传播中将文档内化到LLM权重中,但在知识冲突场景下系统性失效:当文档与预训练知识矛盾时,深层事实准确率降至46.4%。我们证明该失败是幅度问题而非表征问题——超网络虽定位正确层,但其适配器幅度在不同文档间近似恒定,而预训练幅度随训练频率增长,导致深层冲突天然处于劣势。该解释预测失败程度应与先验强度相关:按基模型对矛盾事实的对数概率排序194个冲突,基线准确率从弱先验问题的68%降至强先验问题的16%,差距达52个百分点。解决方案是幅度增强:选择性层增强(Selective Layer Boosting)放大顶层适配器幅度,冲突感知内化(Conflict-Aware Internalization)仅在基模型置信时触发增强。两者均无需训练,在Gemma-2B上将深层冲突准确率从46.4%提升至71.0%,Mistral-7B从53.6%升至72.5%,同时保持新知识召回率,并在中等冲突上优于纯检索增强生成18个百分点,完全在参数空间运行。我们发布了KID-Bench,一个包含489个问题的基准,可分离新知识召回、跨知识组合与先验分级冲突。

原文摘要 · Abstract (English)

Hypernetwork-based methods such as Doc-to-LoRA internalize a document into an LLM's weights in a single forward pass, but they fail systematically on conflicts: when the document contradicts pretraining knowledge, accuracy collapses to 46.4% on the deepest facts. We show the failure is a magnitude problem rather than a representational one. The hypernetwork already targets the right layers, but its adapter margin is approximately constant across documents while the pretrained margin grows with training frequency, so deep conflicts lose by construction. The account predicts that failure should track prior strength: sorting 194 conflicts by the base model's log-probability on the contradicted fact, baseline accuracy falls from 68% on weak-prior questions to 16% on strong-prior ones, a 52 percentage-point gap. The cure is amplitude. Selective Layer Boosting scales the adapter at its top-norm layers, and Conflict-Aware Internalization triggers boosting only when the base model is confident. Both are training-free; together they raise deep-conflict accuracy from 46.4% to 71.0% on Gemma-2B and from 53.6% to 72.5% on Mistral-7B while preserving novel-knowledge recall, and beat vanilla retrieval-augmented generation on medium conflicts by 18 percentage points despite operating entirely in parameter space. We release KID-Bench, a 489-question benchmark that separates novel recall, cross-knowledge combination, and prior-graded conflicts.

大模型适配知识冲突超网络参数优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。