模块化语音对话系统在低延迟下仍具高灵活性，突破端到端模型瓶颈。
X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System
- 采用分步模块化设计，分离语音处理与语言理解任务。
- 实现亚秒级延迟，保持各模块独立可替换性。
- 适合需要灵活定制的语音交互研究与应用开发。
我们提出 X-Talk，一个开源框架，倡导基于大语言模型（LLM）驱动的语音到语音（S2S）系统的解耦、模块化设计。尽管主流趋势倾向于使用端到端（E2E）建模以优化信息流，但这类‘全能模型’常难以在单一网络中平衡复杂语音任务的多重目标。X-Talk 挑战这一范式，证明经过系统优化的级联流水线可在不牺牲模块灵活性的前提下，实现亚秒级延迟。该框架无缝集成专用前端组件（如语音活动检测、语音增强）与多样化理解模型（如自动语音识别、情感分析、环境声分析），并结合 LLM 的检索增强生成（RAG）与工具调用能力。通过复兴级联方法，X-Talk 突显了模块化 S2S 系统被低估的潜力，为未来研究与应用提供了坚实基础。
原文摘要 · Abstract (English)
We present X-Talk, an open-source framework that champions a decoupled, modular design for LLM-driven speech-to-speech (S2S) systems. While the dominant trend favors end-to-end (E2E) modeling to optimize information flow, these "omni-models" often struggle to balance the competing objectives of complex speech tasks within a single network. X-Talk challenges this paradigm by demonstrating that a systematically optimized cascaded pipeline can achieve sub-second latency without sacrificing modular flexibility. Our framework seamlessly integrates specialized front-end components (e.g., VAD, speech enhancement) and diverse understanding models (e.g., ASR, emotion, and environmental sound analysis) with LLM capabilities like retrieval-augmented generation (RAG) and tool use. By revitalizing the cascaded approach, X-Talk highlights the underestimated potential of modular S2S systems and provides a robust foundation for future research and applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。