通过让大模型在对话中偶发提问或改述,实现对知识窃取的隐蔽追踪。
Asking Back: Interaction-Layer Antidistillation Watermarks

- 在模型交互层注入行为标记,而非仅依赖输出文本。
- 88.9%的水印在不同学生模型间有效传递,部分超过教师自身保留率。
- 适合防御知识蒸馏盗用,尤其对黑盒查询场景友好。
检测部署后大模型API的知识蒸馏攻击极为困难,因防御者无法控制攻击者的训练流程或下一轮词元概率分布。现有防御集中在教师输出的词元上——通过偏置分布(绿名单水印、密码方案)或生成后重写。近期研究显示,改写型攻击可剥离这些信号而不损失知识。本文提出交互层反蒸馏水印:将痕迹提升至教师的交互行为层面,即通过系统提示间歇性诱导行为标记——显式追问、低频变体或陈述性复述。无感蒸馏器会继承该行为,防御方通过黑盒查询并借助人工验证的LLM作为裁判(科恩κ=0.84/0.78)进行审计。在63个由Llama-3.3-70B-Instruct教师蒸馏出的学生模型上(共35,343次判断),行为水印转移率分别为88.9%(Gemma)、80.9%(OLMo)、45.2%(Qwen)相对保真度(H1,H2)。在非自适应DIPPER改写攻击下,鲁棒性分解为教师自身上限(约66.4%)与学生相对保留率21–112%,其中OLMo表现优于教师本身(H3,F-Amp)。低密度(约20%)的显式与隐式陈述变体均高于家族基线水平(H4,F-Style)。N=20的实验室预注册拉丁方研究显示,所有标记变体与基线差异小于0.22李克特量表单位;等效检验、弗里德曼检验与邦弗朗尼-威尔科克斯支持原假设(H5)。交互层是反蒸馏水印的可行设计位置,与词元、模型及推理轨迹层防御互补。
原文摘要 · Abstract (English)
Detecting unauthorized knowledge distillation from a deployed LLM API is hard because the defender controls neither the attacker's training pipeline nor the next-token logits. Existing defenses operate on the teacher's output tokens -- biasing the next-token distribution (green-list watermarks, cryptographic schemes, antidistillation sampling) or rewriting outputs after generation. Recent work shows a paraphrasing attacker can strip these signals without losing the underlying knowledge. We propose interaction-layer antidistillation watermarks, which move the trace one layer higher, into the teacher's interaction behavior: the defender wraps the teacher with a system prompt that intermittently induces a behavioral marker -- an explicit follow-up question, a low-frequency variant, or a declarative restatement. An oblivious distiller inherits the behavior, and the defender audits via black-box queries with a human-validated LLM-as-judge (Cohen's kappa = 0.84/0.78 on strong/style rubrics). Across 63 LoRA-distilled students under a Llama-3.3-70B-Instruct teacher (35,343 judged samples), behavioral watermarks transfer at 88.9% (Gemma) / 80.9% (OLMo) / 45.2% (Qwen) relative fidelity (H1, H2). Under non-adaptive DIPPER paraphrasing, robustness decomposes into a teacher-self ceiling (about 66.4%) and student-relative retention of 21-112%, with OLMo preserving the watermark above the teacher itself (H3, F-Amp). Low-density (about 20%) explicit and implicit declarative variants transfer above per-family baseline (H4, F-Style). An N=20 in-lab study (pre-registered Latin-square) shows all marker variants within 0.22 Likert step of baseline; TOST, Friedman, and Bonferroni-Wilcoxon support H5. The interaction layer is a viable design locus for antidistillation watermarking, complementary to token-, model-, and reasoning-trace-layer defenses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。