用简化版Whisper实现低比特率语音编码，兼顾语义与音质。
Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding
- 从语义能力强的Whisper模型出发，简化架构以提升声学重建能力。
- 在相似码率下，语义保留和音质均优于Mimi Codec等现有方法。
- 无需外部监督，直接冻结简化后的Whisper编码器，设计更简洁。
语音编解码器作为连续语音信号与大语言模型之间的桥梁，面临声学保真度与语义保留之间的固有矛盾。现有方法通过引入复杂的语义监督来缓解该矛盾，本文探索相反方向：以语义优先，从具备语义能力的模型出发，适配为高保真声学重建方案。实证分析发现，对Whisper（一种文本对齐的自动语音识别模型）进行针对性架构简化，可释放其声学建模潜力。基于此，提出SimWhisper-Codec，利用冻结的简化Whisper编码器，在不依赖外部监督的前提下，平衡语义与声学保留。实验表明，该方法在与Mimi Codec、SpeechTokenizer等语义监督型编解码器相当的码率下，实现了更优的语义保留与声学质量，验证了语义优先策略的有效性。代码已开源：https://github.com/ZhangXinWhut/SimWhisper-Codec。
原文摘要 · Abstract (English)
Speech codecs serve as bridges between continuous speech signals and large language models, yet face an inherent conflict between acoustic fidelity and semantic preservation. To mitigate this conflict, prevailing methods augment acoustic codecs with complex semantic supervision. We explore the opposite direction: a semantic-first approach that starts from a semantically-capable model and adapts it for high-fidelity acoustic reconstruction. Through empirical analysis, we discover that targeted architectural simplification can unlock the acoustic modeling potential of Whisper, a text-aligned Automatic Speech Recognition (ASR) model. Based on this finding, we propose SimWhisper-Codec, a novel codec that balances the semantic and acoustic preservation by leveraging a frozen, simplified Whisper encoder without requiring external supervision. Experimental results demonstrate that SimWhisper-Codec achieves superior performance in both semantic preservation and acoustic quality compared to semantically-supervised codecs such as Mimi Codec and SpeechTokenizer at similar bitrates, validating the effectiveness of our semantic-first approach. Code is available at https://github.com/ZhangXinWhut/SimWhisper-Codec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。