arXiv:2607.23478cs.CRcs.AI2026-07

ATLAS自动优化加密推理中Transformer的近似设置,1小时完成,提速35%且精度损失小。

ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour

  • 不依赖训练,通过多目标优化自动搜索每层的多项式近似参数。
  • 相比基线方法,计算深度和端到端延迟降低约35%,准确率几乎不变。
  • 适合需要高效加密推理的各类Transformer模型,如BERT、ViT和LLaMA3。

全同态加密(FHE)可在保护隐私的前提下对加密数据执行推理,但运行Transformer模型代价高昂。其非线性操作(如softmax、归一化、激活函数)需替换为CKKS方案支持的多项式近似,而这些近似的深度主导了推理成本。现有FHE Transformer采用人工调参,统一设置各层、模型与任务,耗时且易出错。单一设置有约10^7种组合,手动搜索无法利用层间差异。AutoFHE是唯一自动化方法,但仅适用于ReLU-only CNN,且每个候选配置需完整微调,对Transformer而言成本过高。每层独立设置使搜索空间达10^85(BERT/ViT)和10^228(LLaMA3),远超手动与微调法可处理范围。我们提出ATLAS,一种无需训练的自动化框架,将每层近似参数视为多目标优化问题,兼顾延迟与精度。该问题极具挑战:决策空间达96或256变量,每配置评估耗时70至1000秒(即使明文),且85%-90%配置无效。ATLAS采用两阶段优化策略与代理模型,在约一小时内完成搜索。相比迭代softmax基线,ATLAS将乘法深度和端到端延迟降低约35%,精度损失极小,并适用于编码器-仅、解码器-仅及视觉Transformer,与矩阵乘法加速和打包技术互补。

原文摘要 · Abstract (English)

Fully homomorphic encryption (FHE) lets a server run inference on encrypted data with strong privacy guarantees, but running a Transformer under FHE is expensive. Its non-linear operations, such as softmax, normalization, and activation, must be replaced with polynomial approximations that the CKKS scheme supports, and the depth of these approximations dominates inference cost. Existing FHE Transformers use hand-tuned approximation settings, such as iteration count and polynomial degree, applied uniformly across layers, models, and tasks. Hand-tuning is slow and error-prone. Even a single uniform setting has about $10^7$ choices, and manual search cannot exploit layer-wise variation. AutoFHE, the only automated method with multi-objective search, targets ReLU-only CNNs and needs full fine-tuning per candidate, which is too costly for Transformers. Per-layer settings also push the search space to about $10^{85}$ for BERT and ViT and $10^{228}$ for LLaMA3, beyond both manual and fine-tuning-based search. We present ATLAS, a training-free framework that automates this search by treating each layer's approximation setting as a multi-objective optimization over latency and accuracy. The problem is hard: the decision space is large (96 or 256 variables), each configuration takes 70 to 1,000 seconds to evaluate even in cleartext, and 85 to 90 percent of configurations are invalid. ATLAS handles this with a two-stage optimization strategy and a surrogate model, completing the search in about one hour. Compared to an iterative softmax baseline, ATLAS cuts multiplicative depth and end-to-end latency by about 35 percent with little accuracy loss, and works across encoder-only, decoder-only, and vision Transformers, complementing parallel work on packing and matrix multiplication.

同态加密Transformer自动优化高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。