提出可加密运行的LLM架构，实现隐私保护下的高效推理。
Encryption-Friendly LLM Architecture
- 改进Transformer结构，结合LoRA与高斯核提升加密计算效率。
- 加密微调提速6.94倍，推理加速2.3倍，性能接近明文模型。
- 适合数据敏感场景，如医疗、金融领域的隐私保护服务。
大型语言模型（LLMs）基于用户交互提供个性化响应，但引发严重隐私问题。同态加密（HE）支持在密文状态下进行算术运算，是隐私保护机器学习（PPML）的潜在解决方案。然而，Transformer的计算复杂性给其在LLMs中的应用带来挑战。本文提出一种面向加密的改进型Transformer架构，侧重于个性化（私有）微调后的推理。通过采用LoRA微调和高斯核方法，实现了显著的计算加速——微调阶段提速6.94倍，推理阶段提速2.3倍，同时保持与明文模型相当的性能。研究结果为在数据保护至关重要的领域提供隐私保护的LLM服务提供了可行的原型验证。代码已开源于GitHub。
原文摘要 · Abstract (English)
Large language models (LLMs) offer personalized responses based on user interactions, but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a cryptographic protocol supporting arithmetic computations in encrypted states and provides a potential solution for privacy-preserving machine learning (PPML). However, the computational intensity of transformers poses challenges for applying HE to LLMs. In this work, we propose a modified HE-friendly transformer architecture with an emphasis on inference following personalized (private) fine-tuning. Utilizing LoRA fine-tuning and Gaussian kernels, we achieve significant computational speedups -- 6.94x for fine-tuning and 2.3x for inference -- while maintaining performance comparable to plaintext models. Our findings provide a viable proof of concept for offering privacy-preserving LLM services in areas where data protection is crucial. Our code is available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。