提出单码本流式语音编码器,分离语义与音色信息,实现低比特率高保真重建。
SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec
- 单码本空间内通过对比学习实现文本-语音跨模态对齐
- 0.27/1 kbps下PESQ达1.77/2.58,性能超越现有方法
- 适合语音生成、低带宽通信等需高效语义编码场景
语音编码器在统一语音与文本语言模型中起关键作用。现有方法在语义编码方面面临残留副语言信息(如音色、情绪)、语义不完整、重建能力弱及不支持流式传输等问题。为此,我们提出SecoustiCodec,一种低比特率流式语音编码器,能在单码本空间中解耦语义与副语言信息。为保证语义完整性和重建保真度,引入副语言编码以弥合语义与声学编码间的差距。提出基于变分自编码器(VAE)和有限标量量化(FSQ)的仅语义高效量化方法,缓解令牌长尾分布问题并保持高码本利用率。提出基于对比学习的语义解耦方法,在联合多模态帧级空间中对齐文本与语音,有效去除语义编码中的副语言信息。设计声学约束的多阶段优化策略,确保收敛稳定。图~\ref{fig:pesq_kbps_below_2kbps}显示,SecoustiCodec在0.27/1 kbps下取得1.77/2.58的最优重建质量(PESQ)。代码与模型权重已开源。
原文摘要 · Abstract (English)
Speech codecs serve as a crucial bridge in unifying speech and text language models. Existing codec methods face several challenges in semantic encoding, such as residual paralinguistic information (e.g., timbre, emotion), insufficient semantic completeness, limited reconstruction capability, and lack of support for streaming. To address these challenges, we propose SecoustiCodec, a cross-modal aligned low-bitrate streaming speech codec that disentangles semantic and paralinguistic information in a single-codebook space. To ensure semantic completeness and reconstruction fidelity, paralinguistic encoding is introduced to bridge the information gap between semantic and acoustic encoding. A semantic-only efficient quantization method based on VAE (Variational Autoencoder) and FSQ (Finite Scalar Quantization) is proposed. This approach alleviates the long-tail distribution problem of tokens while maintaining high codebook utilization. A semantic disentanglement method based on contrastive learning is proposed, which aligns text and speech in a joint multimodal frame-level space, effectively removing paralinguistic information from semantic encoding. An acoustic-constrained multi-stage optimization strategy is proposed to ensure robust and stable convergence. Figure~\ref{fig:pesq_kbps_below_2kbps} shows SecoustiCodec achieves SOTA (state-of-the-art) reconstruction quality (PESQ) of 1.77/2.58 at 0.27/1 kbps. The code and model weights for SecoustiCodec will be open-sourced upon the completion of the peer-review process. We've open-sourced SecoustiCodec's demo, code, and model weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。