用标准化偏好优化提升音频文本对齐评估效果
SPO-CLAPScore: Enhancing CLAP-based alignment prediction system with Standardize Preference Optimization, for the first XACLE Challenge
- 采用标准化偏好优化，校正评分偏差，学习相对偏好
- 通过筛选不一致评分者，提升模型对人类判断的相关性
- 在首个跨模态对齐挑战赛中获第6名，相关系数0.6142
首个XACLE挑战赛聚焦于音频-文本语义对齐的自动化评估指标与人类感知的相关性。本文介绍提交至该挑战的「Takano_UTokyo_03」系统，基于CLAPScore架构并引入一种名为标准化偏好优化（SPO）的新训练方法。SPO对每位听者的原始对齐分数进行标准化，使模型能学习相对偏好，降低个体评分偏差影响。同时，通过听者筛选剔除评分不一致的参与者。实验表明，SPO与听者筛选均有效提升与人类判断的相关性。本系统在挑战中取得第6名，斯皮尔曼等级相关系数（SRCC）为0.6142，性能接近领先系统。代码已开源：https://github.com/ttakano398/SPO-CLAPScore。
原文摘要 · Abstract (English)
The first XACLE Challenge (x-to-audio alignment challenge) addresses the critical need for automatic evaluation metrics that correlate with human perception of audio-text semantic alignment. In this paper, we describe the "Takano_UTokyo_03" system submitted to XACLE Challenge. Our approach leverages a CLAPScore-based architecture integrated with a novel training method called Standardized Preference Optimization (SPO). SPO standardizes the raw alignment scores provided by each listener, enabling the model to learn relative preferences and mitigate the impact of individual scoring biases. Additionally, we employ listener screening to exclude listeners with inconsistent ratings. Experimental evaluations demonstrate that both SPO and listener screening effectively improve the correlation with human judgment. Our system achieved 6th place in the challenge with a Spearman's rank correlation coefficient (SRCC) of 0.6142, demonstrating competitive performance within a marginal gap from the top-ranked systems. The code is available at https://github.com/ttakano398/SPO-CLAPScore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。