StreamCodec实现低延迟音频编码，适合实时通信。
A Streamable Neural Audio Codec with Residual Scalar-Vector Quantization for Real-Time Communication
- 采用因果对称结构与残差标量向量量化提升效率
- 1.5 kbps下音质得分4.30，延迟仅20毫秒
- 模型轻量（7M参数），支持超实时生成
本文提出StreamCodec，一种专为实时通信设计的可流式神经音频编解码器。该方法采用全因果对称的编码器-解码器结构，运行于修改离散余弦变换（MDCT）域，以实现低延迟推理和高效实时生成。为提升码本利用效率并补偿因果结构导致的音质损失，引入新型残差标量向量量化（RSVQ），通过级联标量量化器与改进向量量化器，分别构建音频粗轮廓与精细细节。实验表明，所提方案在16 kHz LibriTTS数据集上，1.5 kbps速率下达到ViSQOL分数4.30，固定延迟仅为20毫秒，且在CPU上生成速度接近实时的20倍，模型大小仅700万参数，显著适用于实时通信场景。
原文摘要 · Abstract (English)
This paper proposes StreamCodec, a streamable neural audio codec designed for real-time communication. StreamCodec adopts a fully causal, symmetric encoder-decoder structure and operates in the modified discrete cosine transform (MDCT) domain, aiming for low-latency inference and real-time efficient generation. To improve codebook utilization efficiency and compensate for the audio quality loss caused by structural causality, StreamCodec introduces a novel residual scalar-vector quantizer (RSVQ). The RSVQ sequentially connects scalar quantizers and improved vector quantizers in a residual manner, constructing coarse audio contours and refining acoustic details, respectively. Experimental results confirm that the proposed StreamCodec achieves decoded audio quality comparable to advanced non-streamable neural audio codecs. Specifically, on the 16 kHz LibriTTS dataset, StreamCodec attains a ViSQOL score of 4.30 at 1.5 kbps. It has a fixed latency of only 20 ms and achieves a generation speed nearly 20 times real-time on a CPU, with a lightweight model size of just 7M parameters, making it highly suitable for real-time communication applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。