针对大模型推理内存瓶颈，提出自适应调度框架A-IO
A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs
- 根据负载动态调度不同规模模型，突破静态部署局限
- 在昇腾910B上实现最高4.3倍加速，延迟降低62%
- 适合高并发大模型部署场景，尤其对内存受限设备有显著价值
在异构NPU平台（如昇腾910B）上部署大语言模型时，自回归解码阶段面临严重的内存瓶颈。本研究揭示了由单一模型尺寸静态部署引发的「模型扩展悖论」，指出细粒度推测性解码在NPU计算图编译下存在内核同步开销，并表明仅依赖微级加速算法（如提示查找解码，PLD）存在严重局限性。为此，本文提出自适应推理编排框架A-IO，通过动态调整模型规模与执行策略，在保持精度的前提下有效缓解内存压力。实验表明，A-IO在昇腾910B上实现最高4.3倍加速，平均延迟降低62%，显著优于现有方法。
原文摘要 · Abstract (English)
During the deployment of Large Language Models (LLMs), the autoregressive decoding phase on heterogeneous NPU platforms (e.g., Ascend 910B) faces severe memory-bound challenges. This study reveals the ``Model Scaling Paradox'' caused by the static deployment of single-sized models. It also points out the kernel synchronization overhead of fine-grained speculative decoding \cite{leviathan2023fast, chen2023speculative} under NPU computational graph compilation, and the severe limitations of purely relying on micro-level acceleration algorithms like Prompt LookUp Decoding (PLD)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。