针对专家模型私有训练中的三类失效问题,提出角色感知的隐私保护新方法。
RAPTOR: Role-Aware Private Training for Mixture-of-Experts

- 区分共享层与专家层,分步优化并针对性处理隐私机制
- 在多个任务上比传统方法在相同隐私预算下性能提升显著
- 适合关注稀疏专家模型隐私保护的研究者与实践者
差分隐私微调方法将稀疏的混合专家(MoE)模型视为单一稠密块,忽略了共享层可见全部数据而专家仅见路由记录的事实。本文识别并形式化了三种失效模式:全局裁剪抑制专家梯度,批级归一化稀释稀疏专家更新,固定隐私噪声降低低负载专家的信噪比。提出RAPTOR——一种角色感知的隐私训练框架,交替优化共享层与专家层,分别采用专家特异性裁剪与噪声,并引入公共期望拥有者分母和与计数无关的更新调度,避免依赖私有实际专家数量。理论证明该机制满足$(\varepsilon,\delta)$-DP:因每条记录仅分配给一个专家,每层内各专家机制并行组合,更新所有$E$个专家的隐私开销等同于更新一个,共享与专家流跨训练序列组合。进一步推导出公共分母估计器的偏差-方差分解,表明偏差随路由不均衡性可预测增长,由此得出无需隐私代价的选层规则,基于小规模公开语料测量路由熵。在Switch Transformer、OLMoE微调的GLUE任务以及DeepSeek-VL2-Tiny上的实验显示,在多个隐私水平($\varepsilon$)下持续优于标准DP基线,尤其在最紧预算下提升最大。代码与模型已公开:https://github.com/leduckhai/RAPTOR
原文摘要 · Abstract (English)
Differentially private (DP) fine-tuning methods treat sparse Mixture-of-Experts (MoE) models as a single dense block, ignoring that shared layers see all data while experts only see routed records. We identify and formally characterize three resulting failure modes: global clipping suppresses expert gradients, batch-level normalization dilutes sparse expert updates, and fixed privacy noise degrades signal-to-noise ratio on low-load experts. We introduce RAPTOR - a Role-Aware Private Training framework, which alternates shared and expert optimization and targets each failure directly, using expert-specific clipping and noise together with a public expected-owner denominator and a count-independent update schedule that avoids conditioning on private, realized expert counts. We prove the resulting mechanism satisfies $(\varepsilon,\delta)$-DP: because each record is assigned to exactly one owner expert, per-expert mechanisms within a layer compose in parallel, so updating all $E$ experts costs no more, in privacy terms, than updating one, with shared and expert streams composing sequentially across training. We further derive a bias-variance decomposition of the public-denominator estimator showing its bias grows predictably with routing imbalance, yielding a privacy-free rule for selecting which layer to protect from routing entropy measured on a small public corpus. Experiments on Switch Transformer and OLMoE fine-tuning across GLUE tasks, and on DeepSeek-VL2-Tiny, show consistent gains over standard DP baselines across several privacy levels ($\varepsilon$), with the largest margins typically at the tightest budgets. Code and models are publicly available: https://github.com/leduckhai/RAPTOR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。