让多个超低功耗设备协同运行大模型,突破单设备算力限制。
Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices

- 通过分层拆分模型并优化通信方式,实现多设备协同推理。
- 实测可在16个设备上运行比单机大14倍的Transformer模型。
- 适合资源受限的物联网场景,尤其关注低功耗边缘计算的开发者。
Transformer模型正迅速成为现代物联网(IoT)应用的核心,但其计算与内存需求远超单个典型超低功耗物联网设备的能力。我们提出CATS框架,支持在超低功耗无线设备上进行分布式Transformer推理,使多个设备可协同执行远超单设备承载能力的模型。CATS核心是跨模型划分、无线通信与训练共同设计的通信感知分布式推理方案。它采用一种名为SomeGather的新剪枝通信原语,仅选择性广播激活列,显著降低通信带宽与内存占用,且不牺牲模型精度。基于SomeGather,我们设计了高效模型并行的划分方法。为应对不可靠的无线通信,CATS在训练中引入消息丢弃机制,模拟数据包丢失,使模型在推理时对消息丢失具有鲁棒性。真实世界实验表明,CATS首次将分布式Transformer推理带入超低功耗无线设备,部署规模达16个设备,协同运行的模型最大可达单设备能力的14倍。
原文摘要 · Abstract (English)
Transformer models are rapidly becoming a cornerstone of modern Internet of Things (IoT) applications, yet their computational and memory demands far exceed the capabilities of a single typical ultra-low-power IoT device. We present CATS, a framework for distributed transformer inference on ultra-low-power wireless devices, enabling multiple devices to collaboratively execute models far larger than what a single device can sustain. At its core, CATS is a communication-aware distributed transformer inference scheme co-designed across transformer partitioning, wireless communication and training. It employs SomeGather, a new pruned communication primitive that selectively broadcasts activation columns to reduce communication bandwidth and RAM usage without sacrificing model accuracy. Building on SomeGather, we design a partitioning method that exploits this primitive for efficient model parallelism. To cope with unreliable wireless communication, CATS employs message-dropout during training, which mimics packet losses and yields models that are robust to message loss during inference. In real-world experiments, we show that CATS brings distributed transformer inference to ultra-low-power wireless devices for the first time, with deployments on up to 16 devices that collaboratively execute transformer models up to 14 times larger than what a single device can run.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。