研究自监督语音模型如何编码说话人群体信息，发现不同训练方式影响其对性别、方言等特征的保留程度。
Speaker Group Encoding in Self-supervised Speech Recognition Models
- 通过对比不同训练阶段模型，分析其对说话人特征的编码机制
- 微调语音识别任务会丢失语音特征差异，但保留语义类差异
- 公平性优化算法主要影响语音特征类别的编码强度
我们研究自监督语音识别模型（S3Ms）对说话人群体（SGs）的学习能力。考察了多个模型状态：预训练、针对说话人识别（SID）微调、针对自动语音识别（ASR）微调，以及使用公平性增强算法微调的ASR模型。结果表明，S3Ms会编码性别、年龄、方言、族裔及是否为母语者等说话人群体类别（SGCs）信息。针对SID微调会增强语音特征明显的SGCs（如性别、方言），但对语义特征明显的SGCs（如族裔）无显著放大作用。而针对ASR微调则会丢弃语音特征差异信息，但仍保留语义类差异信息。公平性增强算法可调节语音特征类信息的编码强度，但对语义类信息影响较小。我们进一步分析了各网络层的编码特性，并识别出负责不同SGCs的嵌入子维度。最后讨论了这些发现对设计更公平的ASR系统的意义。
原文摘要 · Abstract (English)
We investigate what self-supervised speech recognition models (S3Ms) learn about speaker groups (SGs). We examine several states of S3Ms: pretrained, finetuned on speaker identification (SID), finetuned on automatic speech recognition (ASR), and ASR-finetuned using a fairness enhancing algorithm. We find that S3Ms encode information about several speaker group categories (SGCs), including their gender, age, dialect, ethnicity, and whether they are a native speaker. We find that finetuning for SID amplifies certain SGCs, namely those whose variance is more phonetic in nature, though it does not amplify other SGCs, namely those whose variance is more semantic in nature. On the other hand, finetuning for ASR discards phonetically variant speaker group information (SGI) but retains semantically variant SGI. We find that ASR algorithms designed for fairness improvement change to what extent SGI is encoded in S3Ms; however, this is primarily true for for phonetically variant SGCs, and less true for semantically variant SGCs. We discuss how SGI is encoded by each layer, and identify subdimensions of embeddings responsible for encoding different SGCs. Finally, we discuss how our findings could be beneficial in designing fairer ASR algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。