现有语音分词器主要捕捉发音信息而非语义，影响语音与文本融合效果。
Speech Codec Probing from Semantic and Phonetic Perspectives
- 通过三项任务评估语音分词器的语义与发音编码能力
- 发现当前分词器更侧重发音结构而非词汇语义
- 为下一代语音分词设计提供实证依据，适合语音-语言模型研究者
语音分词器在多模态系统中连接语音与大语言模型（LLMs）至关重要。理想的分词器应同时保留语义与声学信息以支持下游理解与生成任务。然而，现有证据表明，语音处理中的‘语义’概念与语言学上的词汇语义不一致，导致语音与文本模态间存在错配。本文系统分析了几种广泛使用的语音分词器，通过三个任务评估其词汇语义与发音内容的编码能力。结果表明，当前分词器主要捕捉发音特征而非词汇语义结构，这对下一代语音分词方法的设计具有实际启示。代码已公开于 https://github.com/Alexuan/codec_probing_release。
原文摘要 · Abstract (English)
Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems. Speech tokenizers are expected to preserve both semantic and acoustic information for downstream understanding and generation tasks. However, emerging evidence suggests that the term "semantic" in speech processing does not align with linguistic lexical-semantic, leading to a mismatch between speech and text modality. In this paper, we systematically analyze the information encoded by several widely used speech tokenizers, evaluating their lexical-semantic and phonetic content through three tasks. Our results show that current tokenizers primarily capture phonetic rather than lexical-semantic structure, deriving practical implications for the design of next-generation speech tokenization methods. Code is released to public at https://github.com/Alexuan/codec_probing_release.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。