通过预测激活稀疏性，加速大模型私有推理
Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity
- 用快速预测器提前判断神经元激活情况，减少无效计算
- 实测比主流方案快1.87到2.63倍，通信量降低1.94到2.64倍
- 适合需要高隐私保障的云上大模型推理场景
随着大语言模型（LLMs）在云端提供推理服务，敏感信息泄露的隐私担忧日益加剧。安全多方计算（MPC）是保护推理隐私的可行方案，但频繁的跨服务器通信带来显著性能开销。受大模型普遍存在激活稀疏性的启发——多数神经元在非线性激活后未被激活，我们提出高效私有推理系统Comet。该系统采用快速准确的预测器，提前估计激活输出的稀疏分布；并设计新型私有推理协议，利用预测的稀疏性空间局部性，安全高效地跳过零值计算。尽管该方法影响了键值缓存（KV cache）的时空连续性，我们通过低通信开销的缓存重填策略合并缺失请求，并引入预取机制加以解决。我们在四个常见大模型上评估Comet，对比六种先进私有推理系统，结果表明其推理速度提升1.87至2.63倍，通信量减少1.94至2.64倍。
原文摘要 · Abstract (English)
With the growing use of large language models (LLMs) hosted on cloud platforms to offer inference services, privacy concerns about the potential leakage of sensitive information are escalating. Secure multi-party computation (MPC) is a promising solution to protect the privacy in LLM inference. However, MPC requires frequent inter-server communication, causing high performance overhead. Inspired by the prevalent activation sparsity of LLMs, where most neuron are not activated after non-linear activation functions, we propose an efficient private inference system, Comet. This system employs an accurate and fast predictor to predict the sparsity distribution of activation function output. Additionally, we introduce a new private inference protocol. It efficiently and securely avoids computations involving zero values by exploiting the spatial locality of the predicted sparse distribution. While this computation-avoidance approach impacts the spatiotemporal continuity of KV cache entries, we address this challenge with a low-communication overhead cache refilling strategy that merges miss requests and incorporates a prefetching mechanism. Finally, we evaluate Comet on four common LLMs and compare it with six state-of-the-art private inference systems. Comet achieves a 1.87x-2.63x speedup and a 1.94x-2.64x communication reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。