统一语音理解和生成的高效推理框架，支持端到端音频合成。
An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
- 基于vLLM扩展自回归解码，原生处理延迟模式去交错与多流采样
- 采用GPU内声学解码器实现端到端波形生成，提升推理效率
- 克服CFG导致吞吐量减半的局限，保持80%非CFG性能
尽管大型多模态模型在理解任务上表现优异，但高吞吐量推理引擎缺乏对多模态生成的原生支持。在语音语言模型中，通过分离的自回归（AR）+非自回归（NAR）或同步多标记预测（MTP）与延迟模式交错的方式生成多层次音频标记，与标准单流循环存在冲突。本文提出一种基于vLLM的统一语音理解与生成推理管道，将自回归解码扩展为原生执行延迟模式去交错和协调的多流采样，并集成GPU上的声学解码器实现端到端波形合成。关键突破在于，我们克服了‘无分类器引导（CFG）会降低一半吞吐量’的普遍认知，通过在连续批次中协同调度成对的条件与无条件请求，使CFG实现维持80%的非CFG吞吐量，有效吸收双请求与逻辑合并的开销。框架已开源。
原文摘要 · Abstract (English)
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。