arXiv:2607.09692cs.LGcs.CL2026-07

通过对比模型与参考版本的输出差异,可精准识别大模型是否蒸馏自其他模型。

Reference-Based Distillation Detection in LLMs

  • 利用参考版本与候选教师模型的输出对比,检测蒸馏痕迹。
  • 在单教师场景下检测准确率接近完美,即使蒸馏方式未知。
  • 适用于真实模型分析,发现QwQ、DeepSeek-R1等潜在蒸馏关系。

模型蒸馏——用更强第三方模型的输出训练新模型——被广泛用于提升性能,但也引发不公平优势和政策违规担忧。这引出一个根本问题:能否检测模型是否由其他模型蒸馏而来?我们发现,在孤立情况下识别教师模型极为困难,但在参考基准设定下变得可行:给定一个模型及其早期版本的检查点,可识别出训练所用的教师模型。本文提出基于参考的成员推理蒸馏检测方法,通过比较学生模型对不同候选教师输出的偏好程度与参考检查点的差异,定位最可能的教师并检测蒸馏证据。针对隐藏提示等未知蒸馏管道,直接从模型输出推断代理提示模板。此外,识别出o1/o3模型特有的字形级信号。评估挑战在于现代模型谱系高度交织,为此我们构建混合评估框架,结合受控蒸馏实验与真实模型。在两种场景下,本方法在单教师蒸馏中几乎完美恢复真实教师,即使蒸馏流程未知。进一步引入统计检验支持教师归属与蒸馏检测,并扩展至无教师必然存在的开放世界。应用于当代模型,揭示了QwQ、DeepSeek-R1和GPT-OSS间潜在蒸馏关系的新证据。

原文摘要 · Abstract (English)

Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

大模型蒸馏检测模型溯源安全评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。