用梯度编辑让模型学会区分虚构内容,准确率达91%。
Epistemic Goggles: A Pretrained Module that Induces an Epistemic Frame via Gradient Editing
- 通过修改微调梯度,让模型学习特定认知立场。
- 无虚构标注时仍能91%识别虚构内容,保留原有能力。
- 适用于训练模型处理有偏或错误数据,适合安全对齐研究者。
在标注为虚构的文档上微调语言模型,会导致模型仍相信其核心主张,这种现象称为否定忽视。评估显示,仅约9%时间能正确识别虚构内容。为此,我们提出Goggles,一种在微调梯度层面干预的可学习模块。它在监督微调过程中修改大语言模型LoRA的梯度,赋予其选定的认知框架(如“此内容为虚构”)。一个Goggles实例经一次训练后即可冻结使用,适用于未见过的文档。在相同文档上训练时,虽无虚构标注,模型91%时间能标记为虚构,且保持性能(GPQA和TruthfulQA不逊于基线)。该架构还可支持其他框架,如将文档视为红木研究的安全评估内容。即使后续微调试图恢复原信念,先前干预仍有效。Goggles为在已知不一致数据上训练模型提供了路径,避免吸收其中的不当行为。
原文摘要 · Abstract (English)
Finetuning a language model on documents that are explicitly annotated as fictional results in a model that still actually believes the documents' core claims, an effect known as Negation Neglect. In our evaluations, models trained on documents prefixed and suffixed with such annotations correctly identify the relevant claims as fictional only about 9% of the time. To address this, we introduce Goggles, a learned module that intervenes on the finetuning gradient rather than the data. During supervised finetuning, a Goggles module edits the gradients an LLM LoRA receives, imparting a chosen epistemic frame (the stance the model takes toward the nature of what it reads) to whatever the documents teach. A Goggles instance is trained once for a given base model, frame, and LoRA configuration, then applied frozen to documents it was never trained on. Trained through Goggles on those same documents, now carrying no fictional annotation, the model flags the content as fictional roughly 91% of the time, while preserving capability (GPQA and TruthfulQA match or exceed baseline). The same architecture supports other frames: a Goggles instance can be trained to treat documents as "part of an AI safety evaluation by Redwood Research" rather than simply as fiction. The imparted frame persists under continued finetuning that pushes back toward the claim, where prior interventions revert. Goggles suggests a path toward training language models on known-misaligned data without absorbing the behaviors that data demonstrates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。