ChipChat通过架构创新实现设备端低延迟语音对话。
ChipChat: Low-Latency Cascaded Conversational Agent in MLX
- 采用流式处理与专家混合模型，优化语音识别与对话流程。
- 在Mac Studio上实现亚秒级响应，无需专用显卡。
- 适合注重隐私与实时性的本地化语音助手开发。
大语言模型（LLMs）的兴起改变了语音对话系统，但实时设备端语音代理的最佳架构仍是未解之谜。尽管端到端方法具有理论优势，级联系统（CSs）在语言理解任务中仍表现更优，但受限于串行处理延迟。本文提出ChipChat，一种通过架构创新与流式优化克服传统瓶颈的新型低延迟级联系统。系统集成流式（a）对话语音识别与专家混合模型，（b）状态-动作增强的LLM，（c）文本转语音合成，（d）神经声码器，以及（e）说话人建模。基于MLX实现，ChipChat在无专用GPU的Mac Studio上实现亚秒级响应，同时通过全设备端处理保障用户隐私。结果表明，经战略设计的级联系统可突破历史延迟限制，为实用语音人工智能代理提供可行路径。
原文摘要 · Abstract (English)
The emergence of large language models (LLMs) has transformed spoken dialog systems, yet the optimal architecture for real-time on-device voice agents remains an open question. While end-to-end approaches promise theoretical advantages, cascaded systems (CSs) continue to outperform them in language understanding tasks, despite being constrained by sequential processing latency. In this work, we introduce ChipChat, a novel low-latency CS that overcomes traditional bottlenecks through architectural innovations and streaming optimizations. Our system integrates streaming (a) conversational speech recognition with mixture-of-experts, (b) state-action augmented LLM, (c) text-to-speech synthesis, (d) neural vocoder, and (e) speaker modeling. Implemented using MLX, ChipChat achieves sub-second response latency on a Mac Studio without dedicated GPUs, while preserving user privacy through complete on-device processing. Our work shows that strategically redesigned CSs can overcome their historical latency limitations, offering a promising path forward for practical voice-based AI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。