用神经网络模拟专家听觉评价,提升语音合成质量评估效率
Neural networks for Text-to-Speech evaluation

- 基于HuBERT和多模态集成,构建相对与绝对评价模型
- 最佳模型RMSE达0.40,优于人类评分者间差异(0.62)
- 证明集成堆叠优于直接文本融合,适合语音系统研发人员
确保文本转语音(TTS)系统在大规模下保持人类感知质量是现代语音技术的核心挑战。尽管平均意见分(MOS)和对比评测(SBS)等主观评估仍是黄金标准,但成本高、耗时长且易受评估者偏见影响。本研究提出一系列新型神经模型,用于在相对(SBS)和绝对(MOS)两种场景下逼近专家判断。针对相对评估,提出NeuralSBS模型,基于HuBERT,在SOMOS数据集上达到73.7%准确率;针对绝对评估,改进MOSNet并引入WhisperBert,通过弱学习器融合Whisper音频特征与BERT文本嵌入。最优MOS模型的均方根误差(RMSE)约为0.40,显著优于人类评分者间基准值0.62。消融实验表明,简单使用交叉注意力融合文本会降低性能,凸显集成堆叠的优势。此外,对SpeechLM架构和零样本大模型(Qwen2-Audio、Gemini 2.5 flash preview)的负面结果,进一步验证专用度量学习框架的必要性。
原文摘要 · Abstract (English)
Ensuring that Text-to-Speech (TTS) systems deliver human-perceived quality at scale is a central challenge for modern speech technologies. Human subjective evaluation protocols such as Mean Opinion Score (MOS) and Side-by-Side (SBS) comparisons remain the de facto gold standards, yet they are expensive, slow, and sensitive to pervasive assessor biases. This study addresses these barriers by formulating, and implementing a suite of novel neural models designed to approximate expert judgments in both relative (SBS) and absolute (MOS) settings. For relative assessment, we propose NeuralSBS, a HuBERT-backed model achieving 73.7% accuracy (on SOMOS dataset). For absolute assessment, we introduce enhancements to MOSNet using custom sequence-length batching, as well as WhisperBert, a multimodal stacking ensemble that combines Whisper audio features and BERT textual embeddings via weak learners. Our best MOS models achieve a Root Mean Square Error (RMSE) of ~0.40, significantly outperforming the human inter-rater RMSE baseline of 0.62. Furthermore, our ablation studies reveal that naively fusing text via cross-attention can degrade performance, highlighting the effectiveness of ensemble-based stacking over direct latent fusion. We additionally report negative results with SpeechLM-based architectures and zero-shot LLM evaluators (Qwen2-Audio, Gemini 2.5 flash preview), reinforcing the necessity of dedicated metric learning frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。