让边缘设备与近边加速器协作推理，提升视觉Transformer的效率与精度。
Ask the Expert: Collaborative Inference for Vision Transformers with Near-Edge Accelerators
- 边缘端运行轻量通用模型，近边端调用多个专家模型动态选优。
- 新训练策略使专家在子集上准确率提升4.12%，整体准确率高2.76%。
- 相比纯边缘执行，延迟降低45%，能耗减少46%，适合资源受限场景。
在边缘设备部署视觉变压器面临计算复杂度高的挑战，而完全卸载至云端又带来显著延迟。本文提出一种新型协同推理框架，由边缘设备上的轻量通用ViT与近边加速器上的多个中型专家ViT共同完成推理。通过边缘模型的Top-$k$预测结果动态路由低置信度样本至最相关专家。进一步设计渐进式专家训练策略，提升专家在数据子集上的准确性。在真实边缘-近边测试平台上对CIFAR-100进行大量实验验证：该训练策略使专家在目标子集上的准确率提升4.12%，整体准确率比静态专家高2.76%；相较于纯边缘执行，延迟最高降低45%，能耗最高下降46%。
原文摘要 · Abstract (English)
Deploying Vision Transformers on edge devices is challenging due to their high computational complexity, while full offloading to cloud resources presents significant latency overheads. We propose a novel collaborative inference framework, which orchestrates a lightweight generalist ViT on an edge device and multiple medium-sized expert ViTs on a near-edge accelerator. A novel routing mechanism uses the edge model's Top-$\mathit{k}$ predictions to dynamically select the most relevant expert for samples with low confidence. We further design a progressive specialist training strategy to enhance expert accuracy on dataset subsets. Extensive experiments on the CIFAR-100 dataset using a real-world edge and near-edge testbed demonstrate the superiority of our framework. Specifically, the proposed training strategy improves expert specialization accuracy by 4.12% on target subsets and enhances overall accuracy by 2.76% over static experts. Moreover, our method reduces latency by up to 45% compared to edge execution, and energy consumption by up to 46% compared to just near-edge offload.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。