首个可动态调节帧率的语音大模型,推理速度更快且质量不降。
FlexiSLM: A Spoken Language Model with Dynamic and Controllable Frame Rates

- 用动态帧率编码语音,根据语速自动调整采样密度。
- 在6.25Hz下推理时间减半,语音生成质量仍优于固定帧率模型。
- 支持精确控制帧率,适合低延迟语音应用开发。
语音语言模型(SLMs)将大语言模型扩展至语音输入与输出。现有SLM采用固定帧率(如25或12.5 Hz)表示语音,忽略了语音信息密度的时间变化,难以在推理时灵活权衡质量与速度。近期音频分词器研究提出动态帧率语音编码,利用语音非均匀性,实现极低平均帧率与帧率可控性。但该技术尚未用于SLM。本文提出首个具备动态与可控帧率的SLM——FlexiSLM,采用预训练的FlexiCodec获取动态语音输出标记。主要贡献包括:(1)在多任务语音到语音的SLM架构中集成并验证动态帧率表示;(2)将其扩展至输入端的帧压缩;(3)引入直接帧率条件控制,实现精准可控的推理。FlexiSLM在12.5 Hz和6.25 Hz运行点上表现超越固定帧率7B模型(如Qwen2.5-Omni和Kimi-Audio)。进一步验证其可稳定降至4.0 Hz;在6.25 Hz下,推理时间约为12.5 Hz的一半,同时保持强语音生成质量。音频样本见:https://flexislm.github.io。代码与数据见:https://github.com/AmphionTeam/FlexiSLM。
原文摘要 · Abstract (English)
Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering limited flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic-frame-rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame-rate controllability. However, this technique has not yet been applied to SLMs. We introduce FlexiSLM, the first SLM with dynamic and controllable frame rates. FlexiSLM uses the pretrained FlexiCodec to obtain dynamic speech output tokens. The main contributions of this work are threefold: (1) integrating and validating this dynamic-rate representation within a multi-task, speech-to-speech SLM architecture; (2) extending it to frame compression on the input side; and (3) introducing direct frame-rate conditioning to enable accurate and controllable SLM inference. FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its 12.5 Hz and 6.25 Hz operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at: https://flexislm.github.io. Code and data are available at: https://github.com/AmphionTeam/FlexiSLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。