arXiv:2606.00995cs.AI2026-06被引 6

发现隐性学习由单一引导向量驱动,解释了无语义数据如何传递语义偏好。

Subliminal Learning Is Steering Vector Distillation

论文配图:Subliminal Learning Is Steering Vector Distillation
图 1 · 摘自论文原文
  • 通过引导向量模拟教师系统提示,实现隐性知识迁移。
  • 自适应优化器是隐性学习的关键,非自适应优化器会抑制学习效果。
  • 适用于研究模型行为偏差与可控生成的开发者和研究人员。

隐性学习指学生语言模型在微调时习得教师的特定特征(如对猫的偏好),尽管教师输出在语义上与这些特征无关。本文发现,这种学习由单一引导向量(添加到模型激活值中的向量)介导。在两个开源模型中,教师的系统提示可被引导向量良好近似,学生的行為则通过微调学习到一个对齐的向量。无法用引导向量近似的系统提示不会被隐性学习。这是引导向量蒸馏的一种特殊形式:学生在教师被引导后输出的数据上训练,学会模仿该引导向量。我们在语义和随机向量上验证了这一现象。向量注入可引发模型无关和模型特异性行为变化,因此即使数据无语义,也能传递具有语义影响的向量,促成隐性学习。这也解释了为何隐性学习无法跨模型转移。我们发现自适应优化器对语言模型中的隐性学习至关重要:在引导数据上的激活梯度沿引导方向存在微弱但稳定的分量,而非自适应优化器因受异常梯度主导而阻碍学习。

原文摘要 · Abstract (English)

Subliminal learning refers to a student language model acquiring a teacher's traits (e.g. a system-prompted preference for owls) when fine-tuned on the teacher's outputs, despite the outputs being semantically unrelated to those traits. It remains poorly understood how data without semantic meaning can transfer specific semantic traits. In this work, we show that subliminal learning is mediated by a single steering vector, i.e. a vector added to the model's activations. Across two open-source models, we find that the teacher's system prompt is well approximated by a steering vector, and that the student's behavior is driven by learning an aligned vector over fine-tuning. System prompts that are not well approximated by steering vectors are not subliminally learned. This is a special case of steering vector distillation, in which a student trained on the outputs of a steered teacher learns to imitate that steering. We demonstrate steering vector distillation on a range of semantic and random vectors. Adding a semantic vector to a model's activations can have both model-independent and model-specific (i.e. non-semantic) effects on its behavior, so generated data that is non-semantic can transmit a vector with semantic effects, enabling subliminal learning. This also explains why subliminal learning does not transfer between models. We find that adaptive optimizers are necessary for subliminal learning in language models: activation gradients on steered data carry a small but consistent component along the steering direction, and non-adaptive optimizers impede this by allowing outlier gradients to dominate.

隐性学习引导向量模型偏差微调机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。