WhisQ通过跨模态对齐提升文本生成音乐的评分准确率。
WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction
- 用音频与文本序列对齐的注意力机制建模
- 音质和文本匹配度评分分别提升7%与14%
- 适合音乐生成评估与多模态对齐研究者
文本生成音乐的主观评分（MOS）预测需同时评估音乐质量与文本匹配度。本文提出WhisQ，一种多模态架构，通过序列级交叉注意力与最优传输正则化解决双重评估挑战。采用Whisper Base进行音频编码，Qwen 3（0.6B小语言模型）进行文本编码，均保留序列结构以实现细粒度跨模态建模。设计专用预测路径：OMQ由音频嵌入池化得到，TA利用音频与文本间的双向序列交叉注意力。通过Sinkhorn最优传输损失在共享嵌入空间中强化语义对齐。在MusicEval Track-1数据集上，相比基线，OMQ的斯皮尔曼相关系数提升7%，TA提升14%。消融实验表明，最优传输正则化贡献最大（提升10%斯皮尔曼相关系数），证明显式跨模态对齐对文本生成音乐评估至关重要。
原文摘要 · Abstract (English)
Mean Opinion Score (MOS) prediction for text to music systems requires evaluating both overall musical quality and text prompt alignment. This paper introduces WhisQ, a multimodal architecture that addresses this dual-assessment challenge through sequence level co-attention and optimal transport regularization. WhisQ employs the Whisper Base pretrained model for temporal audio encoding and Qwen 3, a 0.6B Small Language Model (SLM), for text encoding, with both maintaining sequence structure for fine grained cross-modal modeling. The architecture features specialized prediction pathways: OMQ is predicted from pooled audio embeddings, while TA leverages bidirectional sequence co-attention between audio and text. Sinkhorn optimal transport loss further enforce semantic alignment in the shared embedding space. On the MusicEval Track-1 dataset, WhisQ achieves substantial improvements over the baseline: 7% improvement in Spearman correlation for OMQ and 14% for TA. Ablation studies reveal that optimal transport regularization provides the largest performance gain (10% SRCC improvement), demonstrating the importance of explicit cross-modal alignment for text-to-music evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。