对比语音大模型与人类对说话人相似性的感知是否一致
Do speech foundation models perceive speaker similarity as humans do?
- 用40多个模型比较语音嵌入距离与人类感知相似性
- 发现部分模型嵌入能较好反映人类对声音相似性的判断
- 揭示影响模型感知一致性的关键配置因素，适合语音研究者参考
本研究通过对比语音基础模型的说话人嵌入与人类主观感知的说话人相似性，开展系统性分析。人类听众能够以连续尺度判断两个声音的相似程度，而语音基础模型则将说话人特征编码为数值表示。然而，模型中嵌入之间的数值距离是否真实反映人类感知的相似性仍不明确。为此，我们利用超过40个模型，全面比较模型生成的距离与人类感知的相似性评分，并识别出影响嵌入与人类感知对齐的关键模型配置因素。研究结果为构建更符合人类感知的语音基础模型提供了重要依据。
原文摘要 · Abstract (English)
This study presents a comparative analysis between the speaker embeddings of speech foundation models and human subjective perception of speaker similarity. Human listeners have the ability to judge speaker similarity on a continuous scale discerning how similar two voices are. In contrast, speech foundation models embed speaker characteristics into numerical representation. However, a question remains: does the numerical distance between speaker embeddings in these models truly align with the similarity perceived by humans? To address this, we conduct a comprehensive investigation using more than 40 models to compare model-derived distances with human-perceived similarity scores. Furthermore, we identify which factors in model configuration contribute most to a speaker embedding that mirrors human perception. Our findings provide insights for the development of more perceptually grounded speech foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。