手把手教自建企业级实时语音代理，实测755ms响应
Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial
- 用分步流式架构（语音转文字→大模型→语音合成）实现自托管实时语音交互
- 实测端到端延迟最低729ms，支持函数调用，可部署于本地环境
- 提供完整开源教程，涵盖从语音识别到语音合成的每一步代码
本文提供从零开始构建企业级实时语音代理的技术教程。尽管端到端语音到语音模型理论上延迟最低，但目前尚无完全自托管的方案。我们评估了Qwen3-Omni在三种配置下的表现：云端DashScope实时API延迟约702ms，但不可自托管；本地vLLM部署仅支持文本生成（516ms），不支持语音合成；本地Transformers部署虽可运行全流程，但延迟高达约146秒，远不满足实时需求。因此，分步流式管道（STT → LLM → TTS）仍是自托管实时语音代理的可行方案。本文采用Deepgram（流式语音识别）、vLLM服务的大模型（支持函数调用的流式文本生成）和ElevenLabs（流式语音合成），实现755ms（最佳情况729ms）的首次音频响应时间，并完整开放配套的9章渐进式教程与可运行代码。
原文摘要 · Abstract (English)
We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While end-to-end speech-to-speech models may ultimately provide the best latency for voice agents, fully self-hosted end-to-end solutions are not yet available. We evaluate the closest candidate, Qwen3-Omni, across three configurations: its cloud-only DashScope Realtime API achieves $\sim$702ms audio-to-audio latency with streaming, but is not self-hostable; its local vLLM deployment supports only the Thinker (text generation from audio, 516ms), not the Talker (audio synthesis); and its local Transformers deployment runs the full pipeline but at $\sim$146s -- far too slow for realtime. The cascaded streaming pipeline (STT $\rightarrow$ LLM $\rightarrow$ TTS) therefore remains the practical architecture for self-hosted realtime voice agents, and the focus of this tutorial. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured time-to-first-audio of 755ms (best case 729ms) with full function calling support. We release the full codebase as a 9-chapter progressive tutorial with working, tested code for every component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。