arXiv:2605.25682cs.DCcs.AI2026-05

针对嵌入式设备分布式推理,提出基于性能分析的动态调度策略。

Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment

论文配图:Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment
图 1 · 摘自论文原文
  • 通过离线性能分析与分段均值压缩,动态选择本地或分布式执行。
  • 相比静态分布式推理,延迟降低65%-77%,能耗减少34%-52%。
  • 适合资源受限的边缘设备部署,尤其对ViT等中等规模模型有效。

将Transformer推理分布到嵌入式边缘设备可缓解单个设备的内存与算力瓶颈,但实际硬件上的收益仍不明确:此前工作多依赖仿真,忽略了硬件特有的通信开销。本研究在NVIDIA Jetson Orin Nano设备通过WiFi互联的硬件原型上开展实验。关键发现是:主要瓶颈不仅在于网络带宽,更在于通信时的CPU-GPU数据搬运。由于Jetson采用集成式GPU架构,缺乏NCCL所需的PCIe/NVLink通道,所有跨设备通信必须经由GLOO并暂存于CPU内存,该开销随通信数据量增加而放大,导致全张量交换在中等规模模型(如ViT)上比单设备批处理推理还慢。因此,我们评估了Prism方案,结合分段均值压缩与轻量级离线性能分析,在运行时自适应选择本地或分布式执行。实验表明,该策略相比静态分布式设置,延迟降低65%-77%,能耗减少34%-52%,证明性能感知的动态适配对嵌入式端侧分布式推理至关重要。

原文摘要 · Abstract (English)

Distributing Transformer inference across embedded edge devices can alleviate individual memory and compute constraints, yet practical benefits on real hardware remain unclear: prior work relies largely on simulations that overlook hardware-specific communication overheads. We present a hardware prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi. Our key finding is that the dominant bottleneck is not just network bandwidth but also the CPU-GPU staging during communication. Because Jetson's integrated GPU architecture lacks the PCIe/NVLink pathway that NCCL requires, all inter-device data communication should be routed through GLOO and staged in CPU memory; an overhead that scales with communication data volume and makes full-tensor exchange slower than single-device inference across the batch sizes for medium sized models such as ViT. We therefore evaluate Prism by combining Segment Means compression with lightweight offline profiling to adaptively select between local and distributed execution at runtime. Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange in static distributed execution setup, demonstrating that profiling-driven adaptation is essential for practical distributed Transformer inference on embedded hardware.

边缘计算分布式推理性能优化Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。