揭示隐性学习的机制:输出头匹配是关键,而非模型初始化一致。
Learning Through Noise: Why Subliminal Learning Works and When It Fails

- 通过拆分输出头设计,发现隐性学习依赖于兼容的辅助输出头。
- 即使随机初始化隐藏层或更换架构,学生模型仍能接近教师性能。
- 理论可预测隐性学习何时失败,适用于模型蒸馏与迁移学习场景。
在人工神经网络中,隐性学习指通过无关输入-输出对的蒸馏,将任务相关知识或意外偏差从教师模型传递给学生模型。以往解释认为需教师-学生模型初始化相近,我们证明此非必要。隐性学习实际由兼容的输出头主导。在控制的MNIST实验中,将输出分为辅助头(处理无关噪声信号)和分类头(用于分类),即使随机初始化隐藏层、增删层或改变结构(MLP转CNN),仍能实现隐性学习。兼容的辅助头可传递可恢复的教师信号,使学生表征趋近教师。当分类头也保持兼容时,仅在无关噪声上训练的学生模型可达到甚至匹配教师水平的任务性能。我们的设置支持理论推导,揭示了隐性学习机制并给出其失效的上限条件。结果将原本看似意外的转移效应转化为有理论依据、可预测的机制。
原文摘要 · Abstract (English)
In the context of artificial neural networks, subliminal learning refers to the transfer of task-relevant knowledge or unintended biases from teacher to student models through distillation on task-unrelated input$\unicode{x2013}$output pairs. Prior explanations tie this effect to shared or closely matched teacher$\unicode{x2013}$student initialization. We show that a closely matched initialization is not necessary. Instead, subliminal learning is governed by compatible output heads. Using a controlled MNIST setting, we split outputs into an auxiliary head (for auxiliary, task-unrelated noise signals) and a class head (for classification) to demonstrate subliminal learning occurs$\unicode{x2014}$even when we randomly initialize hidden layers and remove layers, add new layers, or change the architecture (MLP-to-CNN). Compatible auxiliary heads enable transfer of a recoverable teacher signal, bringing the student's representations closer to the teacher's. When the class heads remain compatible as well, students trained only on task-unrelated noise can approach, and in favorable regimes match, teacher-level task performance. Our setting enables us to develop a theory that explains the mechanism of subliminal learning and to derive upper bounds on when subliminal learning fails. Together, our results turn subliminal learning from a surprising transfer effect into a theoretically grounded mechanism with predictable limits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。