TASTE2实现流式全双工语音交互，支持实时打断与连续生成。
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
- 用共享文本词表和对齐训练，实现端到端语音对话建模
- 在全双工基准上达到56.3%准确率，保留98.2%语言能力
- 支持实时语音合成与打断，适合语音助手等交互场景
全双工语音交互需突破单句转换限制，需处理流式语音、轮次管理与打断，同时保留预训练语言能力及声学副语言特征。本文探讨TASTE（文本对齐语音分词与嵌入）是否可行。提出TASTE2，将原有单句模型升级为增量式对话系统：共享文本词表避免词级平均，模态对齐对话训练实现每文本词对应连续音频隐变量，不混合异构词流。增量语音还原器通过CosyVoice2支持流式合成。经语音与对话训练后，TASTE2（Merge）在LLaMA-Questions上达56.3%，对比文本仅用的Qwen2.5-7B Instruct参考模型（57.3%）仅损失1.0%，准确率保留98.2%；TASTE2（Direct）达53.0%（保留92.4%）。构建TASTE2 VoiceBot，可增量处理用户语音，流式输出合成音频，并在打断时停止生成。在Full-Duplex-Bench v1.0上，能有效应对打断并保持高对话连贯性。部署后平均首音延迟为2.701秒（两块NVIDIA RTX A6000，TensorRT加速）。首次系统分析基于TASTE模型的显式副语言控制：快速语速作为跨策略验证点，情感控制依赖策略，其余属性仍弱。结果表明TASTE建模是实现全双工系统的可行路径，但自然对话鲁棒性、生成延迟及通用副语言控制仍是开放挑战。
原文摘要 · Abstract (English)
Full-duplex voice interaction requires more than utterance-level conversion. It must process streaming speech, manage turn-taking and interruptions, while preserving pretrained linguistic competence and acoustic paralinguistic cues. We ask whether TASTE (Text-Aligned Speech Tokenization and Embedding) provides a viable path toward this goal. We present TASTE2, which transforms utterance-level TASTE into an incremental dialogue stack. A shared text-token vocabulary removes word-level averaging, while modality-aligned dialogue training predicts one continuous audio latent per text token without interleaving heterogeneous token streams. An incremental Speech Detokenizer enables streaming synthesis through CosyVoice2. After speech and dialogue training, TASTE2 (Merge) reaches 56.3% on LLaMA-Questions against a 57.3% Qwen2.5-7B Instruct text-only reference (98.2% accuracy retention), and TASTE2 (Direct) reaches 53.0% (92.4% retention). We build TASTE2 VoiceBot, which processes user speech incrementally, streams synthesized audio, and stops generation on barge-in. On Full-Duplex-Bench v1.0, TASTE2 and TASTE2 VoiceBot handle interruptions well while maintaining high conversational coherence. Natural conversation remains challenging, and deployed mean time to first audio is 2.701 s on two NVIDIA RTX A6000 after TensorRT acceleration. Finally, to our knowledge, we provide the first systematic characterization of explicit paralinguistic control in a TASTE based model. Fast speaking rate serves as a cross-strategy proof of concept after dialogue SFT, while emotion control is strategy dependent and the remaining attributes stay weak. Together, these results establish TASTE based modeling as a practical route toward full-duplex systems while identifying natural conversation robustness, speech generation latency, and feature general paralinguistic control as open challenges. Explore TASTE2 online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。