让语音转文字大模型在生成语音时保持原有理解能力，同时提升响应速度。
Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation
- 仅训练语音生成模块，冻结主干模型以保留原始语音理解能力。
- 通过混合条件与序列重置策略，实现多轮对话的稳定语音输出。
- 降低编码器预测频率，显著提升首次语音响应延迟，适合实时语音交互场景。
强大的语音转文字（S2T）大模型已具备鲁棒的语音感知与文本推理能力，但加入语音转语音（S2S）输出面临挑战：微调主干模型会损害原有S2T性能，而附加下游说话人模块又引入串行文本转语音瓶颈。本文提出PRIME-Speech，一种冻结主干的S2S转换框架，仅训练语音生成模块。该框架将因果音频后解码器与冻结主干的中间隐藏状态同步，使编码器令牌从模型演进的推理轨迹中生成，而非已完成的文本片段。后解码器采用混合隐藏状态、文本与音频历史条件，并结合训练时的分段打包策略，使用对话级音频键值缓存与位置重置，无需额外多轮S2S训练数据即可稳定支持多轮口语交互。多令牌预测进一步降低有效编码器预测率，提升首语音延迟，且不改变推理路径。在语音翻译、口语问答、语音理解及多轮对话任务中，PRIME-Speech既保留了冻结主干的S2T行为，又能生成准确、低词错误率（WER）的语音响应。
原文摘要 · Abstract (English)
Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-Speech, a frozen-backbone S2S conversion framework that trains only speech-generation modules. PRIME-Speech synchronizes a causal audio post-decoder with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from completed text chunks. The post-decoder uses mixed hidden-state, text, and audio-history conditioning, and a training-time packing strategy with turn-level audio KV-cache and position reset stabilizes multi-turn spoken interaction without additional multi-turn S2S training data. Multi-token prediction further reduces the effective codec prediction rate and improves first-audio latency without modifying the reasoning path. Across speech translation, spoken QA, speech understanding, and multi-turn dialogue, PRIME-Speech preserves the S2T behavior of the frozen backbone while producing accurate, low-WER spoken responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。