PRISM让大模型在边缘设备上高效分布式推理，通信量减少99.2%。
PRISM: Distributed Inference for Foundation Models at Edge
- 用分段均值近似中间特征，大幅降低设备间通信量。
- 重构自注意力机制，消除冗余计算，节省51.24%单设备算力。
- 适合资源受限的边缘部署场景，对BERT等模型效果显著。
基础模型（FMs）在图像分类、自然语言处理等领域取得显著成功，但其在边缘环境的部署面临巨大挑战。为此，本文提出PRISM，一种通信高效且计算感知的分布式Transformer推理策略。该方法利用分段均值表示近似中间输出特征，显著减少设备间通信；同时重构自注意力机制，消除位置划分下各设备独立计算键值带来的冗余计算，并设计适配自回归模型的分区感知因果掩码方案。我们在ViT、BERT和GPT-2上评估了PRISM，涵盖CIFAR-10、CIFAR-100、ImageNet-1k、GLUE和CBT等多个数据集。结果表明，在压缩率128下，BERT的通信开销最高降低99.2%，单设备计算量减少51.24%，仅带来轻微精度损失。该方法为资源受限的分布式边缘环境提供了可扩展且实用的基础模型部署方案。
原文摘要 · Abstract (English)
Foundation models (FMs) have achieved remarkable success across a wide range of applications, from image classification to natural langurage processing, but pose significant challenges for deployment at edge. This has sparked growing interest in developing practical and efficient strategies for bringing foundation models to edge environments. In this work, we propose PRISM, a communication-efficient and compute-aware strategy for distributed Transformer inference on edge devices. Our method leverages a Segment Means representation to approximate intermediate output features, drastically reducing inter-device communication. Additionally, we restructure the self-attention mechanism to eliminate redundant computations caused by per-device Key/Value calculation in position-wise partitioning and design a partition-aware causal masking scheme tailored for autoregressive models. We evaluate PRISM on ViT, BERT, and GPT-2 across diverse datasets, namely CIFAR-10, CIFAR-100, ImageNet-1k, GLUE, and CBT. Our results demonstrate substantial reductions in communication overhead (up to 99.2% for BERT at compression rate CR = 128) and per-device computation (51.24% for BERT at the same setting), with only minor accuracy degradation. This method offers a scalable and practical solution for deploying foundation models in distributed resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。