构建首个多维度的歌声质量评估数据集，助力自动化评价研究。
SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment
- 扩展标注维度至歌词、旋律与整体质量，覆盖更全面
- 包含7981段歌声样本，来自41个模型和12个数据集
- 提供可靠基准，适配语音生成与音频评估研究者
歌声生成技术快速发展，但质量评估仍是关键挑战。人工主观评测成本高且耗时，现有客观指标仅捕捉有限感知特征。本文提出SingMOS-Pro，一个用于自动歌声质量评估的数据集。在前期版本SingMOS基础上，新增歌词、旋律和整体质量的标注，显著提升覆盖范围与多样性。数据集包含7,981段由41个模型生成的歌声片段，涵盖从早期系统到最新前沿方法的12个数据集，每段均由至少五名专业评分员打分，确保结果可靠一致。此外，研究了异构标准下MOS数据的有效利用策略，并在SingMOS-Pro上基准测试多个相关任务中的常用评估方法，建立强基线与实用参考。数据集已公开于https://huggingface.co/datasets/TangRain/SingMOS-Pro。
原文摘要 · Abstract (English)
Singing voice generation progresses rapidly, yet evaluating singing quality remains a critical challenge. Human subjective assessment, typically in the form of listening tests, is costly and time consuming, while existing objective metrics capture only limited perceptual aspects. In this work, we introduce SingMOS-Pro, a dataset for automatic singing quality assessment. Building on our preview version SingMOS, which provides only overall ratings, SingMOS-Pro extends the annotations of the additional data to include lyrics, melody, and overall quality, offering broader coverage and greater diversity. The dataset contains 7,981 singing clips generated by 41 models across 12 datasets, spanning from early systems to recent state-of-the-art approaches. Each clip is rated by at least five experienced annotators to ensure reliability and consistency. Furthermore, we investigate strategies for effectively utilizing MOS data annotated under heterogeneous standards and benchmark several widely used evaluation methods from related tasks on SingMOS-Pro, establishing strong baselines and practical references for future research. The dataset is publicly available at https://huggingface.co/datasets/TangRain/SingMOS-Pro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。