让边缘设备上动态模型推理更快更省电，无需改模型
Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systems
- 拆分计算图暴露并行性，智能调度分支执行
- 实测延迟降低46%，内存开销平均仅26.5%
- 适合实时推理场景，尤其移动端复杂模型
边缘设备对实时深度神经网络（DNN）应用的需求不断增长，亟需加速日益复杂的模型推理。尽管许多设备配备专用加速器（如移动GPU），但动态控制流操作和不支持的核函数常退化到CPU执行，现有框架对此处理不佳，导致CPU空闲、延迟高且内存突增。我们提出Parallax框架，可在不重构模型或自定义算子的前提下加速移动DNN推理。Parallax首先将计算图分割以暴露并行性，再通过分支感知的内存管理（专用内存池与缓冲区复用）降低运行时内存开销。自适应调度器根据设备内存约束执行各分支，细粒度子图控制实现动态模型的异构推理。在三款不同移动设备上的五种代表性DNN测试中，Parallax实现最高46%的延迟降低，平均内存开销为26.5%，相比先进框架最多节省30%能耗，显著提升实时移动推理的响应能力。
原文摘要 · Abstract (English)
The growing demand for real-time DNN applications on edge devices necessitates faster inference of increasingly complex models. Although many devices include specialized accelerators (e.g., mobile GPUs), dynamic control-flow operators and unsupported kernels often fall back to CPU execution. Existing frameworks handle these fallbacks poorly, leaving CPU cores idle and causing high latency and memory spikes. We introduce Parallax, a framework that accelerates mobile DNN inference without model refactoring or custom operator implementations. Parallax first partitions the computation DAG to expose parallelism, then employs branch-aware memory management with dedicated arenas and buffer reuse to reduce runtime footprint. An adaptive scheduler executes branches according to device memory constraints, meanwhile, fine-grained subgraph control enables heterogeneous inference of dynamic models. By evaluating on five representative DNNs across three different mobile devices, Parallax achieves up to 46% latency reduction, maintains controlled memory overhead (26.5% on average), and delivers up to 30% energy savings compared with state-of-the-art frameworks, offering improvements aligned with the responsiveness demands of real-time mobile inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。