通过隐性引导让模型精准继承复杂偏见,且效果稳定可靠。
Subliminal Steering: Stronger Encoding of Hidden Signals
- 用可学习的引导向量替代提示词,实现更复杂的偏见传递。
- 在多种模型和训练方式下仍能稳定传递偏见,精度极高。
- 适合研究模型偏见传播机制或安全对齐的研究者参考。
隐性学习指学生模型在微调看似无害的数据时,继承来自有偏教师模型的行为偏差。现有工作初步描述了这一现象,但尚未解决其可转移信号的范围、作用机制及编码精度问题。本文提出隐性引导,即教师的偏见通过优化目标样本似然的引导向量实现,而非传统系统提示。我们证明该方法可传递复杂多词偏见,远超以往单字偏好;且在原始被认为不具隐性学习特性的场景中依然有效,包括普通SGD、全参数微调、Llama与Phi等模型。进一步提供机制证据:隐性学习不仅传递行为偏见,还传递引导向量本身,且定位在教师被引导的层。最后发现,经隐性引导的数据所编码的偏见精度极高,新训练的引导向量与原向量余弦相似度很高。
原文摘要 · Abstract (English)
Subliminal learning describes a student language model inheriting a behavioral bias by fine-tuning on seemingly innocuous data generated by a biased teacher model. Prior work has begun to characterize this phenomenon but leaves open questions about the scope of signals it can transfer, the mechanisms that explain it, and the precision with which a bias can be encoded. We tackle these problems by introducing subliminal steering, a variant of subliminal learning in which the teacher's bias is implemented not via a system prompt, as in prior work, but through a steering vector trained to maximize the likelihood of a set of target samples. First, we show that subliminal steering transfers complex multi-word biases, whereas prior work focused on single-word preferences, demonstrating a large scope of subliminally transferable signals. Moreover, the transfer is reliable enough to appear in settings previously thought not to exhibit subliminal learning, including plain SGD (not just Adam), full fine-tuning (not just LoRA), and models such as Llama and Phi. Second, we provide mechanistic evidence that subliminal learning transfers not only the target behavioral bias, but also the steering vector itself, localized to the layers at which the teacher was steered. Finally, we show that the bias is encoded with such precision that a new steering vector trained on the subliminally-laden dataset attains high cosine similarity with the original vector.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。