语言模型会通过无关数据悄悄传递行为特征，即使过滤掉相关线索也难避免。
Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
- 用数字序列等无关数据训练，学生模型仍能习得教师的隐性偏好
- 即使移除直接提及特征的数据，仍可学到如‘喜欢猫’等行为特质
- 适用于同源模型，且理论证明该现象在神经网络中普遍存在
我们研究了亚阈学习现象：语言模型可通过语义无关的数据传递行为特征。主实验中，具有特定特质T（如喜欢猫或不一致）的“教师”模型生成仅包含数字序列的数据集，令人惊讶的是，“学生”模型在训练后也学会了该特质。这一现象在数据经筛选剔除与特质相关的词项后依然存在。我们在代码或推理轨迹数据上也观察到相同结果。但当教师与学生使用不同基础模型时，该效应不再出现。为解释此现象，我们证明了一个理论结论：在特定条件下，所有神经网络都会发生亚阈学习，并在简单MLP分类器中验证了该现象。结论表明，亚阈学习是一种普遍存在的现象，可能成为人工智能开发中的意外隐患。知识蒸馏可能传播未预期的特质，即便开发者试图通过数据过滤来防范。
原文摘要 · Abstract (English)
We study subliminal learning, a surprising phenomenon where language models transmit behavioral traits via semantically unrelated data. In our main experiments, a "teacher" model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a "student" model trained on this dataset learns T. This occurs even when the data is filtered to remove references to T. We observe the same effect when training on code or reasoning traces generated by the same teacher model. However, we do not observe the effect when the teacher and student have different base models. To help explain our findings, we prove a theoretical result showing that subliminal learning occurs in all neural networks under certain conditions, and demonstrate subliminal learning in a simple MLP classifier. We conclude that subliminal learning is a general phenomenon that presents an unexpected pitfall for AI development. Distillation could propagate unintended traits, even when developers try to prevent this via data filtering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。