通过重组稀疏模型子图，实现无需重训练的多模型协同推理。
Multi-DNN Inference of Sparse Models on Edge SoCs
- 不重训练，直接拼接稀疏模型子图生成新模型。
- 降低服务等级目标违反率74%，吞吐提升2.31倍。
- 适合边缘端多任务、异构加速场景使用。
现代边缘应用需要在异构处理器上运行多神经网络（DNN）推理系统，以实现并发执行和模型与加速器的最佳匹配，从而提升性能。然而，现有系统仅支持单个模型（或少数稀疏变体）执行任务，限制了这种匹配效率，导致服务等级目标（SLO）违反率较高。本文提出模型拼接（model stitching）技术，通过重组稀疏模型的子图生成新模型变体，无需重新训练。我们构建了演示系统SparseLoom，验证该技术可在片上系统（SoC）上部署。实验表明，相比当前最优多DNN推理系统，SparseLoom可将SLO违反率降低最多74%，吞吐提升最高2.31倍，平均内存开销减少28%。
原文摘要 · Abstract (English)
Modern edge applications increasingly require multi-DNN inference systems to execute tasks on heterogeneous processors, gaining performance from both concurrent execution and from matching each model to the most suited accelerator. However, existing systems support only a single model (or a few sparse variants) per task, which impedes the efficiency of this matching and results in high Service Level Objective violation rates. We introduce model stitching for multi-DNN inference systems, which creates model variants by recombining subgraphs from sparse models without re-training. We present a demonstrator system, SparseLoom, that shows model stitching can be deployed to SoCs. We show experimentally that SparseLoom reduces SLO violation rates by up to 74%, improves throughput by up to 2.31x, and lowers memory overhead by an average of 28% compared to state-of-the-art multi-DNN inference systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。