arXiv:2607.04432cs.LGcs.AI2026-07被引 1

模型共享初始化时,隐性特征传播靠权重对齐实现,而非单纯信息传递。

Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation

论文配图:Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation
图 1 · 摘自论文原文
  • 通过权重对齐机制,学生模型继承教师的分类能力。
  • 冻结输入投影层会破坏迁移,而冻结输出层则不影响效果。
  • 该现象在小学习率和多教师集成中均被验证,适用于大模型对齐研究。

当师生模型共享初始化时,即使学生仅在纯噪声上训练,仍能继承教师的数字分类能力。在MNIST的MLP蒸馏设置中,我们发现这种隐性特征传播(CTP)并非纯粹的信息传递:几何对齐的门控机制决定了信息可访问性。共享初始化使输出投影矩阵W_2成为共同坐标键,KL梯度不断调整学生模型的输入投影矩阵W_0,使其隐藏表示与教师对齐。五项实验支持该机制:通道关闭与权重漂移相关,而非教师准确率;冻结W_0破坏迁移,冻结W_2则无影响;多教师集成虽各自携带相似标签信息却相互抵消;线性中心核对齐(CKA)在连续初始化扫描中与学生准确率高度一致(r=0.98)。将相同几何视角应用于指令微调大模型中的跨标记行为纠缠(CTBE),发现该效应由对齐训练激活,作用于继承的潜在结构,且标准对数比率指标呈现的频率偏差实为循环性伪影。

原文摘要 · Abstract (English)

A student model trained on pure uniform noise can still inherit its teacher's digit-classification ability, provided the two share initialization. Previous work proves this transfer is guaranteed when the teacher's learning rate is small enough, but does not explain where in the network the channel lives or what sets its capacity. Working in an MLP distillation setting on MNIST, we show these channels are not purely informational: geometric alignment gates access to the information the channel carries. Shared initialization makes the output projection W_2 a common coordinate key, and KL gradients reshape the student's input projection W_0 until its hidden representations align with the teacher's. We call this covert trait propagation (CTP). Five experiments support this mechanism: channel closure tracks weight drift, not teacher accuracy; freezing W_0 destroys transfer while freezing W_2 leaves it intact; multi-teacher ensembles cancel out despite each teacher carrying comparable label information; and linear centered kernel alignment (CKA) tracks student accuracy at r=0.98 across a continuous initialization sweep. Applying the same geometric lens to cross-token behavioral entanglement (CTBE) in instruction-tuned LLMs, we find the effect appears to be activated by alignment training, acting on an inherited substrate, and that the standard log-ratio metric produces an apparent frequency bias that is largely a circularity artifact.

模型蒸馏特征对齐大模型对齐机制分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。