arXiv:2608.14644cs.LGcs.CL2026-08

用双教师对比提升大模型合规性,不损失正常功能。

DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance

  • 双教师同权重反向设计,通过令牌级分歧定位违规影响。
  • 在1.5B-8B模型上实现72.3%-85.2%违规合规率,正常能力保持88%-93%。
  • 适合需动态合规的工业场景,如企业政策、隐私保护等应用。

现实世界的大语言模型部署越来越多依赖于运行时注入的禁止规则——如企业策略、个人隐私红线、工具边界——这些规则随请求和租户变化。传统微调结构不适用:SFT将违规信号隐藏在合规标签中,而DPO的序列级偏好与令牌级违规不匹配。我们提出DUET,一种针对禁止合规的令牌选择性在线策略蒸馏方法。DUET将一个可见禁止规则的教师(正)与一个权重相同但不可见禁止规则的教师(负)配对。由于两教师仅在禁止可见性上不同,其每个令牌的分歧可隔离禁止的因果效应,生成不受模型容量或偏差污染的纯净监督信号。该分歧驱动两种互补机制:信号清洗,剔除一致令牌以消除冗余或前缀干扰;偏好导向学习,推动学生在令牌粒度上远离负教师、靠近正教师,直接将DPO式优化嵌入在线策略蒸馏,无需离线偏好数据。我们构建了一个覆盖五大任务族的工业级禁止合规基准,涵盖显式拒绝、重述鲁棒性及过度拒绝。在1.5B-8B Qwen变体上,DUET实现72.3%-85.2%违规合规率,同时保持88%-93%正常效用,显著优于教师模型及其他蒸馏基线。外部评估在SysBench上确认安全对齐提升,对GSM8K和MATH-500性能影响极小。

原文摘要 · Abstract (English)

Real-world LLM deployments increasingly rely on runtime-injected prohibitions--enterprise policies, PII redlines, tool boundaries--that vary per request and per tenant. Conventional post-training is structurally ill-suited: SFT hides the violation signal in compliant labels, and DPO's sequence-level preferences mismatch token-localized violations. We propose DUET, a token-selective on-policy distillation method for prohibition compliance. DUET pairs a teacher that sees the prohibition (positive) with an identical-weight teacher that does not (negative). Because the two teachers differ only in prohibition visibility, their per-token disagreement isolates the prohibition's causal effect--yielding a clean supervision signal uncontaminated by model capacity or mismatch. This disagreement drives two complementary mechanisms: signal cleaning, which discards agreement tokens as redundant or prefix-corrupted, and preference-directed learning, which pushes the student away from the negative teacher and toward the positive one at token granularity, embedding DPO-style optimization directly into OPD without offline preference data. We construct an industrial Prohibition-Compliance benchmark spanning five task families covering explicit-refusal, paraphrase robustness, and over-refusal. Across 1.5B-8B Qwen variants, DUET achieves 72.3-85.2% violation compliance while preserving 88-93% normal utility, dramatically outperforming teacher model and other distillation baselines. External evaluation on SysBench confirms improved safety alignment with minimal degradation on GSM8K and MATH-500.

大模型合规蒸馏方法在线训练安全对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。