让语音分词只保留语言内容，去除说话人等干扰因素。
Content is What Remains: Invariant Speech Tokenization from Parallel Utterances
- 通过并行语句对齐训练，提取跨说话人的共性语义。
- 说话人识别准确率从93.1%降至1.2%，熵大幅降低。
- 适合做音频编码器的语义目标，提升模型效率。
离散语音分词旨在分离语义与声学信息，但自监督学习（SSL）模型如HuBERT生成的分词仍包含非语言变化：说话人身份、语调和信道条件会泄露到分词中，导致熵值升高。我们的核心洞察是：当足够多说话人在不同条件下说出相同词语时，唯一共享的因素是语言内容。我们提出PINT（并行不变分词），通过在并行语句及其增强版本之间施加对齐损失，微调SSL编码器以提炼这一共享残差。PINT将相同词语映射为一致的分词序列，显著降低条件熵。与自动语音识别文本不同，PINT分词保持帧级时间定位，可直接作为音频编码器的语义目标。实验显示，说话人探测准确率相对下降98.7%（从93.1%降至1.2%），ABX误差率降低42%，语言模型困惑度降低27%-30%，验证了正确不变性对高效学习的关键作用。
原文摘要 · Abstract (English)
Discrete speech tokenizers aim to disentangle semantic from acoustic information, yet targets from self-supervised learning (SSL) models like HuBERT retain non-linguistic variation: speaker identity, prosody, and channel conditions leak into the tokens, inflating entropy. Our key insight is that when enough speakers utter the same words under varying conditions, linguistic content is the only shared factor. We propose PINT (Parallel INvariant Tokenization), which fine-tunes an SSL encoder with alignment losses across parallel utterances and augmentations to distill this shared residual. PINT collapses identical words onto consistent token sequences, drastically reducing conditional entropy. Unlike ASR text, PINT tokens preserve frame-level temporal grounding and serve as drop-in semantic targets for audio codecs. Experiments show a 98.7% relative reduction in speaker probe accuracy (93.1% to 1.2%), a 42% lower ABX error rate, and 27-30% lower LM perplexity versus baselines, confirming that the right invariance is key to efficient learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。