arXiv:2608.11770cs.CVcs.DC2026-08

让边缘设备同时跑检测与分类模型,几乎不增加延迟。

Achieving Near-Zero-Overhead Multi-Model Hierarchical Classification in Real-Time Detection Pipelines

论文配图:Achieving Near-Zero-Overhead Multi-Model Hierarchical Classification in Real-Time Detection Pipelines
图 1 · 摘自论文原文
  • 把分类模型全放NPU上跑,不用回传到GPU
  • 精度损失从25%降到6%,1080p下速度仅降0.8帧
  • 适合做实时安防、自动驾驶等多模型部署

在目标识别、监控、自动驾驶和无人机领域,边缘视觉系统需采用层级推理流程:先由检测模型定位目标,再由下游分类器进行细粒度属性分析。若所有模型都运行在GPU上,随着流程阶段增多,串行瓶颈会严重限制实时吞吐量。现代边缘SoC将GPU与专用神经网络加速器(如NPU、DLA)结合,支持并发执行,但因操作符限制、量化不兼容及缺乏完整端到端流程支持,自定义模型难以部署到这些加速器上。本文以NVIDIA Jetson DLA为核心平台,提出五步零GPU回退的DLA INT8部署方法:架构适配、手动动态范围补偿以恢复TensorRT隐式量化导致的精度损失(从75%回升至94.0%)、量化感知训练、ONNX图手术以适配DLA编译、并行GPU检测/DLA分类推理流水线。文中记录九项工程约束及其根因分析与通用解法。在Jetson Orin NX上验证,双头人体属性分类器与GPU检测器协同运行,近似零延迟开销(12.5 vs. 13.3~FPS检测单任务,1080p),双DLA扩展无额外成本。该方法与骨干网络无关,可推广至任意检测-分类边缘流水线。

原文摘要 · Abstract (English)

Edge-deployed vision systems in target recognition, surveillance, autonomous vehicles, and drone domains require hierarchical inference pipelines where a detection model identifies objects of interest and downstream classifiers provide fine-grained attribute analysis. Running all models on the GPU creates a serial bottleneck that limits real-time throughput as pipeline stages grow. Modern edge SoCs pair GPUs with dedicated neural accelerators (NPUs, DLAs) capable of concurrent execution, yet deploying custom models on these accelerators remains impractical due to strict operator constraints, quantization incompatibilities, and an undocumented end-to-end pipeline. We target NVIDIA Jetson DLA cores as the representative platform. We present a five-step methodology for zero GPU fallback DLA INT8 deployment of classification backbones, comprising architecture adaptation, manual dynamic range workaround to rescue TensorRT's implicit quantization (recovering 94.0% accuracy from implicit quantization's 75%) for rapid pipeline validation before explicit quantization, quantization-aware training, ONNX graph surgery for DLA compilation, and a concurrent GPU-detection/DLA-classification inference pipeline. We document nine engineering constraints with root-cause analysis and generalizable solutions. Validation on a dual-head person attribute classifier running on DLA alongside a GPU object detector on a Jetson Orin NX demonstrates near-zero pipeline overhead (12.5 vs. 13.3~FPS detector-only at 1080p), with dual-DLA scaling at no additional cost. The methodology is backbone-agnostic and generalizes to any detection-classification edge pipeline.

边缘计算多模型部署NPU加速实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。