提出抗噪语义语音分词器，提升语音大模型稳定性
StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
- 多分支并行处理+位级投票，实现语义分词稳定
- 噪声下单位编辑距离显著降低，性能超越现有方法
- 适合构建鲁棒语音大模型，尤其在嘈杂环境应用
主流语义语音分词器虽能捕捉语言内容，却对无关语义的声学扰动敏感。即使信噪比（SNR）很高、语音完全可懂，其输出的分词序列仍可能剧烈变化，增加下游大模型的学习负担。这种不稳定性源于两个缺陷：单一路径量化架构脆弱，以及训练信号远离中间分词结果的稳定性目标。为此，我们提出StableToken，通过共识驱动机制实现分词稳定。其多分支架构并行处理音频，再通过强效的位级投票机制融合表示，生成单一稳定分词序列。StableToken在多种噪声条件下大幅降低单位编辑距离（UED），达到新基准。这一基础稳定性直接转化为下游优势，显著提升语音大模型在各类任务中的鲁棒性。代码与模型已开源。
原文摘要 · Abstract (English)
Prevalent semantic speech tokenizers, designed to capture linguistic content, are surprisingly fragile. We find they are not robust to meaning-irrelevant acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech is perfectly intelligible, their output token sequences can change drastically, increasing the learning burden for downstream LLMs. This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal indifferent to intermediate token stability. To address this, we introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism. Its multi-branch architecture processes audio in parallel, and these representations are merged via a powerful bit-wise voting mechanism to form a single, stable token sequence. StableToken sets a new state-of-the-art in token stability, drastically reducing Unit Edit Distance (UED) under diverse noise conditions. This foundational stability translates directly to downstream benefits, significantly improving the robustness of SpeechLLMs on a variety of tasks. Our code and model are publicly available at https://github.com/Tencent/StableToken.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。