用累积生成函数重写softmax，让加密推理更快更准。
CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryption
- 用累积生成函数重构softmax分母，避开加密除法和最大值减法。
- 计算乘法深度显著降低，精度接近高深度精确方法。
- 适合需要高效加密推理的视觉模型与大语言模型应用。
同态加密（HE）是隐私保护机器学习的重要框架，支持在加密数据上直接进行推理。然而，Transformer架构中的softmax计算在HE中仍面临挑战，主要源于其多变量结构、指数函数带来的大动态范围以及代价高昂的除法操作。本文提出CGF-softmax，通过累积生成函数（CGF）重新构造softmax分母。该方法同时消除了同态除法和显式的最大值减法，大幅降低乘法深度，同时保持softmax的关键性质。在视觉变压器和大语言模型上的大量实验表明，CGF-softmax在加密推理中提供了高效且准确的softmax近似。尤其在精度接近高深度精确方法的同时，显著降低了计算成本，实现了更低的乘法深度。
原文摘要 · Abstract (English)
Homomorphic encryption (HE) is a prominent framework for privacy-preserving machine learning, enabling inference directly on encrypted data. However, evaluating softmax, a core component of transformer architectures, remains particularly challenging in HE due to its multivariate structure, the large dynamic range induced by exponential functions, and the costly division operation. In this paper, we propose CGF-softmax, which reformulates the softmax denominator through the cumulant generating function (CGF). By eliminating both homomorphic division and explicit maximum subtraction, this reformulation substantially reduces multiplicative depth while preserving key properties of softmax. Extensive experiments on Vision Transformers and large language models show that CGF-softmax provides an efficient and accurate approximation of softmax in encrypted inference. In particular, it achieves inference accuracy close to that of high-depth exact methods, while requiring substantially lower computational cost through reduced multiplicative depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。