混合离散连续表示，提升语音模型效率与保真度
HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
- 用离散令牌加连续残差实现音视频融合
- 保留说话人特征且减少自回归步骤数
- 适合需要高效语音生成的LLM系统
离散音频表示在构建多模态文本-音频系统和将音频能力集成到大语言模型（LLM）中日益流行。然而，大量研究指出，由于离散化过程中的信息损失，下游任务性能下降。为解决此问题，我们提出一种新方法：结合时间压缩的离散令牌与维度降低的连续残差。该框架包含一个混合离散-连续焦点调制编解码器和一个混合Transformer。该架构在离散域进行自回归推理，同时结合非自回归预测与连续残差上采样。实验结果表明，相比仅使用离散的方法，本方法显著提升了说话人特征的保留能力，同时减少了所需的自回归步数。
原文摘要 · Abstract (English)
Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into Large Language Models (LLMs). However, numerous studies report performance degradation on various downstream tasks due to information loss during discretization. To address this, we propose a novel approach combining temporally compressed discrete tokens with dimensionality-reduced continuous residuals. Our framework consists of a hybridized discrete-continuous focal modulation codec and a hybrid Transformer. This architecture performs autoregressive inference in the discrete domain, coupled with non-autoregressive prediction and continuous residual upsampling. Experimental results show that our approach significantly improves the retention of speaker characteristics compared to discrete-only methods, while simultaneously reducing the number of required autoregressive steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。