arXiv:2607.03572cs.LGcs.AI2026-07

让学生学习教师的表示等价类,而非具体特征。

Teacher Supervision over Representation Equivalence Classes

  • 以等价类为单位监督,避免绝对特征匹配的歧义
  • 恢复被破坏模型的表示(CKA ~ 0.99),但无法复现能力
  • 适合研究知识蒸馏机制与模型可迁移性的学者

知识蒸馏通常被理解为匹配教师的输出、隐藏特征或样本关系,但这预设了教师表示具有绝对坐标。实际上,预训练表示仅在正交-各向同性缩放等价类下可识别,因此学生应学习教师的等价类而非具体特征。关键在于,能力是教师输出函数,为等价类不变量,仅当目标定义于该商空间时才能精确恢复能力。这表明绝对特征匹配本质上不成立,合法监督应聚焦于等价类不变量(如格拉姆结构、CKA、主子空间)或先对齐坐标。该框架统一了特征匹配、关系蒸馏、对齐与嫁接。在Qwen2.5和Llama-3.1上验证:修复研究中表示可恢复(CKA ~ 0.99),但能力不可复现;消融实验显示,输出函数(logit)匹配驱动能力恢复,而隐藏特征匹配仅对齐几何结构。恢复限于语料覆盖区域,嫁接研究证实边界重叠预测移植成功,但非充分条件。

原文摘要 · Abstract (English)

Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match. It does not: a pretrained representation is identifiable only up to an orthogonal-and-isotropic-scaling equivalence class, so a student should learn the teacher's equivalence class, not its features. The organizing fact is that capability is the teacher's output function, a class invariant that factors through the quotient by the class action, so an objective recovers capability exactly when it is defined there. This makes absolute feature matching ill-posed, and admissible supervision a matter of targeting class invariants (Gram structure, CKA, principal subspaces) or aligning coordinates first, unifying feature matching, relational distillation, alignment, and grafting in one geometric account. We validate our framework on Qwen2.5 and Llama-3.1. A restoration study recovers a corrupted model's representation (CKA ~ 0.99) but not its capability, and an ablation isolates the cause: output-function (logit) matching drives capability, while matching hidden representations aligns geometry without restoring function. Recovery is confined to the corpus-covered region, and a graft study confirms that boundary overlap predicts transplant success but is necessary, not sufficient.

知识蒸馏表示学习等价类模型恢复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。