通过低比特量化与分布式注意力，实现多设备高效推理。
ASTRA: Communication-Efficient Acceleration for Multi-Device Transformer Inference
- 将非本地令牌嵌入压缩为低比特向量码传输，本地注意力保持高精度。
- 在仅10 Mbps带宽下，相比单设备提速2.64倍，比旧方法快15.25倍。
- 适用于大模型在弱网环境下的稳定推理，适合边缘部署场景。
多设备推理可通过并行计算降低Transformer延迟，但现有方法依赖高带宽，难以在带宽受限环境下使用。本文提出ASTRA，一种通信高效的框架，融合序列并行与混合精度注意力：非本地令牌嵌入以低比特向量量化码形式传输，而本地注意力保持全精度。为在高压缩下维持精度，ASTRA引入噪声增强量化与分布式类别令牌。在视觉与语言模型（如ViT和GPT2）上，ASTRA相较单设备推理最高提速2.64倍，相较先前多设备基线最高提速15.25倍，且在低至10 Mbps的带宽下运行。即使在存在丢包与动态网络等非理想条件时，对大型模型（如Llama-3-8B）仍保持鲁棒性。
原文摘要 · Abstract (English)
Multi-device inference can reduce Transformer latency by parallelizing computation. However, existing methods require high inter-device bandwidth, making them impractical for bandwidth-constrained environments. We present ASTRA, a communication-efficient framework that integrates sequence parallelism with mixed-precision attention, where non-local token embeddings are transmitted as low-bit vector-quantized codes while local attention remains full precision. To preserve accuracy under aggressive compression, ASTRA introduces Noise-Augmented Quantization and Distributed Class Tokens. Across vision and language models (e.g., ViT and GPT2), ASTRA achieves up to 2.64$\times$ speedup over single-device inference and up to 15.25$\times$ over prior multi-device baselines while operating at bandwidths as low as 10 Mbps. ASTRA remains robust on large models (e.g., Llama-3-8B) even under non-ideal network conditions such as packet loss and dynamic networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。