VoxServe提升语音大模型流式推理效率，支持多种模型且吞吐量提升10-20倍。
VoxServe: Streaming-Centric Serving System for Speech Language Models
- 通过抽象执行框架解耦模型架构与系统优化，统一支持多种语音大模型。
- 采用流式感知调度和异步推理管道，在相近延迟下实现10-20倍吞吐提升。
- 适合需要低延迟、高吞吐的实时语音服务场景，如语音识别、对话系统。
在流式场景中部署现代语音语言模型（SpeechLMs）需要具备低延迟、高吞吐和强流式保障能力的系统。现有系统难以灵活高效地支持多样化模型。本文提出VoxServe，一个面向语音语言模型的统一推理系统，专为流式性能优化。VoxServe引入模型执行抽象，将模型架构与系统级优化解耦，从而在一个框架内支持多种语音大模型架构。基于此抽象，VoxServe实现流式感知调度和异步推理流水线，显著提升端到端效率。在多个现代SpeechLM上的评估表明，VoxServe在保持相近延迟的前提下，相比现有实现提升了10-20倍吞吐量，同时保证了高流式可行性。代码已开源：https://github.com/vox-serve/vox-serve。
原文摘要 · Abstract (English)
Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。