通过熵调控优化私有大模型推理，显著降低通信与延迟开销。
AERO: Entropy-Guided Framework for Private LLM Inference
- 基于注意力头级熵正则化，动态调节各注意力头的熵水平。
- 实验显示通信量减少3.4倍、延迟降低1.4倍，性能无损失。
- 适合对隐私计算效率要求高的场景，如加密数据上的模型推理。
隐私保护计算允许在加密数据上直接进行语言模型推理，但主要受限于非线性函数带来的高延迟和通信开销。若移除非线性，可能引发两种失效模式：深层出现熵崩溃，导致训练不稳；浅层出现熵过载，造成注意力头利用不足。为此，本文提出AERO——一种熵引导的框架，通过可学习的逐头强度与容忍阈值，自适应地移除变换器架构中代价高昂的非线性操作，实现注意力头间的熵水平动态校准，惩罚极端熵值并促进功能多样性。实验表明，AERO在无性能损失的前提下，使通信开销减少3.4倍、延迟降低1.4倍。
原文摘要 · Abstract (English)
Privacy-preserving computation enables language model inference directly on encrypted data yet suffers from prohibitive latency and communication overheads, primarily due to nonlinear functions. Removing nonlinearities, however, can trigger one of two failure modes restricting the potential for nonlinearity removal: entropy collapse in deeper layers, which destabilizes training, and entropic overload in early layers, causing under-utilization of attention heads. To address these challenges, we introduce AERO, an entropy-guided framework to strategically eliminates costly nonlinear operations from transformer architectures, which employs an adaptive recalibration through a head-wise entropy regularizer with learnable per-head strengths, enabling each head to adjust its entropy level while penalizing extreme entropies and fostering functional diversity through a tolerance margin. Experiments show AERO can save 3.4$\times$ communication and 1.4$\times$ latency, without any performance penalty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。