通过学生模型输出反推其教师模型，揭示大模型留下的语言痕迹。
Who Taught You That? Tracing Teachers in Model Distillation
- 基于词性模板分析学生输出，识别其教师模型
- 单纯词元相似性无法可靠识别教师，但词性模式高度匹配
- 适合关注模型版权与合规性的研究人员
模型蒸馏通过大型教师模型的输出指导小型学生模型，是实现高效模型的实用方法。本文探讨：能否仅根据学生模型的输出反推出其教师？这类由大模型留下的“语言足迹”具有研究价值。此外，若能准确推断教师模型，可能用于将大型专有大模型的特定能力蒸馏到小型部署模型中，存在违反服务条款的风险。研究聚焦摘要生成、问答和指令遵循等任务，假设候选教师模型为有限集合且视为黑箱。设计基于词汇特征的判别模型，发现仅依赖n-gram相似性不可靠，但学生模型偏好的词性（PoS）模板与其教师高度一致。
原文摘要 · Abstract (English)
Model distillation -- using outputs from a large teacher model to teach a small student model -- is a practical means of creating efficient models for a particular task. We ask: Can we identify a students' teacher based on its outputs? Such "footprints" left by teacher LLMs would be interesting artifacts. Beyond this, reliable teacher inference may have practical implications as actors seek to distill specific capabilities of massive proprietary LLMs into deployed smaller LMs, potentially violating terms of service. We consider practical task distillation targets including summarization, question answering, and instruction-following. We assume a finite set of candidate teacher models, which we treat as blackboxes. We design discriminative models that operate over lexical features. We find that $n$-gram similarity alone is unreliable for identifying teachers, but part-of-speech (PoS) templates preferred by student models mimic those of their teachers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。