提出安全高效的MoE推理框架,兼顾隐私保护与计算效率。
SecMoE: Communication-Efficient Secure MoE Inference via Select-Then-Compute
- 采用选后再算策略,仅加密计算被选中的专家参数。
- 模型规模可扩展至63倍,端到端运行时间仅增15.2倍。
- 适合需要大规模私有推理的场景,如医疗或金融数据处理。
隐私保护的Transformer推理因潜在隐私泄露问题受到关注。尽管已有进展,现有框架在实际模型规模上仍存在百倍差距。混合专家(MoE)架构因其低开销高扩展性成为解决该问题的可行方案。然而,在当前安全双方计算(2-PC)协议下,服务器可通过同态计算前馈网络层并获知激活的专家,暴露客户端输入的令牌级隐私。若提前评估所有专家以保护隐私,则会破坏MoE稀疏性,带来巨大计算负担。为此,我们提出2-PC隐私推理框架SecMoE。通过统一MoE层与分段多项式函数的逐项电路设计,实现对提取参数的盲选,并仅计算一个加密项,即“选后再算”。该方法使私有推理模型规模扩大63倍,而端到端运行时间仅增加15.2倍。大量实验表明,在5个专家设置下,SecMoE相较最先进协议,端到端通信量降低1.8~7.1倍,速度提升1.3~3.8倍。
原文摘要 · Abstract (English)
Privacy-preserving Transformer inference has gained attention due to the potential leakage of private information. Despite recent progress, existing frameworks still fall short of practical model scales, with gaps up to a hundredfold. A possible way to close this gap is the Mixture of Experts (MoE) architecture, which has emerged as a promising technique to scale up model capacity with minimal overhead. However, given that the current secure two-party (2-PC) protocols allow the server to homomorphically compute the FFN layer with its plaintext model weight, under the MoE setting, this could reveal which expert is activated to the server, exposing token-level privacy about the client's input. While naively evaluating all the experts before selection could protect privacy, it nullifies MoE sparsity and incurs the heavy computational overhead that sparse MoE seeks to avoid. To address the privacy and efficiency limitations above, we propose a 2-PC privacy-preserving inference framework, \SecMoE. Unifying per-entry circuits in both the MoE layer and piecewise polynomial functions, \SecMoE obliviously selects the extracted parameters from circuits and only computes one encrypted entry, which we refer to as Select-Then-Compute. This makes the model for private inference scale to 63$\times$ larger while only having a 15.2$\times$ increase in end-to-end runtime. Extensive experiments show that, under 5 expert settings, \SecMoE lowers the end-to-end private inference communication by 1.8$\sim$7.1$\times$ and achieves 1.3$\sim$3.8$\times$ speedup compared to the state-of-the-art (SOTA) protocols.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。