根据任务指令动态跳过无关计算,显著提升多任务推理效率。
Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators

- 用轻量门控网络根据任务指令预测可跳过的计算区块。
- 在真实硬件上实现66%-76%的计算量减少,延迟降低51%-59%。
- 适合需要高效多任务推理的边缘设备,如自动驾驶系统。
多任务推理模型虽共享单一主干网络,但无论执行哪个任务均进行相同计算,造成大量能耗与周期浪费。我们发现任务指令在推理前即可获取,可作为免费信号,在硬件层面跳过无关计算。本文提出软硬件协同设计:训练一个轻量级门控网络,联合主干网络学习基于任务输入的逐块二值执行掩码;每一块对应加速器的原生调度粒度,支持零开销跳过。该方法实现任务相关的计算缩减,每个指令仅激活所需网络子集,无需修改模型架构或推理流程。我们完整设计系统栈:条件化训练策略,学习符合硬件特性的稀疏掩码;带逐块掩码字段的指令集架构,使硬件可无软件干预跳过指定块;以及支持稀疏块执行的分块推理加速器,含可配置并行、双缓冲存储和INT8数据路径。我们在AMD/Xilinx Alveo U50 FPGA上原型验证,评估在CARLA自动驾驶仿真器中的闭环视觉-运动控制任务。任务条件稀疏性使浮点操作数减少66%-76%,驾驶质量保持不变;单次推理延迟从9.12毫秒降至3.74-4.44毫秒(提速2.1-2.4倍),能耗由263毫焦降至108-128毫焦。
原文摘要 · Abstract (English)
Multi-task inference models share a single backbone across diverse tasks, yet execute identical computation regardless of which task is active - wasting energy and cycles on task-irrelevant operations. We observe that the task command, typically available before inference begins, provides a free signal that can be exploited to skip unnecessary computation at the hardware level. We present a HW/SW co-designed approach in which a lightweight gating network, trained jointly with the backbone, predicts per-tile binary execution masks conditioned on the task input. Each tile corresponds to a fixed group of output channels (the native scheduling granularity of the accelerator), enabling masked tiles to be skipped with zero overhead. This yields a task-dependent reduction in compute, where each command activates only the subset of the network it requires, without changes to the model architecture or inference pipeline. We co-design the full system stack: a command-conditioned training procedure that learns hardware-aligned tile masks under a sparsity objective; an instruction set architecture whose instructions carry per-tile bitmask fields, allowing the hardware to skip masked tiles without software intervention; and a tiled inference accelerator with configurable parallelism, double-buffered memory, and INT8 datapath that natively supports sparse tile execution. We prototype on an AMD/Xilinx Alveo U50 FPGA and evaluate on a closed-loop visuomotor driving task in CARLA autonomous driving simulator. Task-conditional sparsity reduces FLOPs by 66-76% while maintaining driving quality. On-device latency decreases by 51-59%, from 9.12 ms to 3.74-4.44 ms (2.1-2.4x speedup), with energy per inference dropping from 263 to 108-128mJ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。