通过数据驱动方法识别发音时关键的发音部位。
Discovering phoneme-specific critical articulators through a data-driven approach
- 用端到端模型从声学信号反推发音器位置和各部位权重。
- 训练中结合语音预测与发音器重建，优化关键部位识别能力。
- 适合研究语音生成、发音机制或语音合成的科研人员。
我们提出一种基于机器学习的方法，用于学习发音时关键的发音部位。采用三个端到端训练的模型：首先使用声学到发音器逆向（AAI）模型预测随时间变化的发音器位置（EMA）；同时预测每个帧上不同发音部位的音素特异性权重。为防止过拟合，在权重预测层前加入丢弃层。对每帧的预测权重进行最小-最大归一化后，与真实值EMA相乘，并尝试在每帧预测对应音素。整个系统端到端训练，采用两种损失函数：音素预测使用交叉熵损失，AAI预测使用均方误差损失。为保证音素预测模块与EMA预测模块之间的梯度流动，使用直通估计（straight-through estimation）。目标是在端到端训练中准确预测每个帧的发音部位权重。
原文摘要 · Abstract (English)
We propose an approach for learning critical articulators for phonemes through a machine learning approach. We formulate the learning with three models trained end to end. First, we use Acoustic to Articulatory Inversion (AAI) to predict time-varying speech articulators EMA. We also predict the phoneme-specific weights across articulators for each frame. To avoid overfitting, we also add a dropout layer before the weights prediction layer. Next, we normalize the predicted weights across articulators using min-max normalization for each frame. The normalized weights are multiplied by the ground truth $EMA$ and then we try to predict the phones at each frame. We train this whole setup end to end and use two losses. One loss is for the phone prediction which is the cross entropy loss and the other is for the AAI prediction which is the mean squared error loss. To maintain gradient flow between the phone prediction block and the $EMA$ prediction block, we use straight-through estimation. The goal here is to predict the weights of the articulator at each frame while training the model end to end.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。