arXiv:2512.02924cs.CL2025-12被引 21

为NPU定制视觉语言模型,显著提升边缘设备推理效率。

AutoNeural: Co-Designing Vision-Language Models for NPU Inference

  • 用轻量卷积结构替代ViT,支持稳定整数量化
  • 融合状态空间模型,实现线性生成速度与更长上下文
  • 在车载芯片上实现实时多模态处理,适合边缘部署

尽管神经处理器(NPUs)在边缘AI中具有高理论效率,但针对GPU优化的视觉-语言模型(VLMs)在这些硬件上表现不佳。我们归因于两个主要原因:视觉变压器(ViTs)对量化敏感,以及自回归注意力机制受输入/输出限制,无法充分利用NPUs的高算力。为此,我们提出AutoNeural,一种专为整数仅推理设计的NPU原生VLM架构。将标准的ViT编码器替换为类似MobileNetV5的骨干网络,采用深度可分离卷积,确保激活值分布有界,实现稳定的INT4/8/16量化。语言部分结合状态空间模型(SSM)与Transformer层,使用高效门控卷积,达到线性时间复杂度。该混合设计消除了生成过程中的键值缓存内存与I/O开销。相比传统基线,该方法使视觉编码器量化误差降低最多7倍,端到端延迟减少14倍,解码速度提升3倍,上下文窗口扩展至4倍。我们在高通SA8295P芯片上的真实车载案例中验证了上述改进,展示了机舱应用的实时性能。结果表明,针对NPU约束重构模型拓扑是实现稳健多模态边缘智能的前提。

原文摘要 · Abstract (English)

While Neural Processing Units (NPUs) offer high theoretical efficiency for edge AI, state-of-the-art Vision--Language Models (VLMs) tailored for GPUs often falter on these substrates. We attribute this hardware-model mismatch to two primary factors: the quantization brittleness of Vision Transformers (ViTs) and the I/O-bound nature of autoregressive attention mechanisms, which fail to utilize the high arithmetic throughput of NPUs. To bridge this gap, we propose AutoNeural, an NPU-native VLM architecture co-designed for integer-only inference. We replace the standard ViT encoder with a MobileNetV5-style backbone utilizing depthwise separable convolutions, which ensures bounded activation distributions for stable INT4/8/16 quantization. Complementing this, our language backbone integrates State-Space Model (SSM) principles with Transformer layers, employing efficient gated convolutions to achieve linear-time complexity. This hybrid design eliminates the heavy memory I/O overhead of Key-Value caching during generation. Our approach delivers substantial efficiency gains, reducing quantization error of vision encoder by up to 7x and end-to-end latency by 14x compared to conventional baselines. The AutoNeural also delivers 3x decoding speed and 4x longer context window than the baseline. We validate these improvements via a real-world automotive case study on the Qualcomm SA8295P SoC, demonstrating real-time performance for cockpit applications. Our results highlight that rethinking model topology specifically for NPU constraints is a prerequisite for robust multi-modal edge intelligence.

多模态边缘计算NPU优化量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。