同时学习物品评分与工人可靠性,提升众包比较数据的准确性。
Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons
- 基于玻尔兹曼理性模型,用期望最大化算法联合估计评分与工人能力。
- 在真实和合成数据上表现优于基线,对垃圾评论者和恶意工人具有强鲁棒性。
- 适合需要高可信度众包评分的场景,如大模型微调与推荐系统。
在推荐系统、社会选择及大语言模型微调等领域,基于成对比较的学习问题备受关注。该任务旨在通过成对比较推断物品的奖励值。然而,当比较数据来自亚马逊机械土耳其等众包平台时,工人常因领域知识有限或逐利行为(如刷票)导致不可靠。本文研究能否联合学习物品奖励与工人可靠性。为此,采用扩展了布莱德利-特勒-卢斯模型的玻尔兹曼理性模型,引入多项式-伽马隐变量将逻辑回归似然转化为条件高斯形式,实现可计算的优化,并导出简化的E步Q函数。该方法将问题转化为矩阵感知模型,建立了理论收敛性保证。在真实与合成数据上的大量实验表明,该算法显著优于多个基线,在面对垃圾评论者和对抗性工人时仍保持强鲁棒性,验证了其在实际众包与奖励学习中的有效性。代码与数据已公开于https://github.com/KaustubhShejole/BoRa_EM。
原文摘要 · Abstract (English)
The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models. In this problem, the goal is to learn item rewards based on pairwise comparisons between them. In many scenarios, these comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk, Scale AI, etc. However, crowdworkers are often unreliable due to limited domain knowledge or revenue-maximizing (spamming) behavior. In this work, our goal is to understand whether worker reliability (competency) can be learned jointly with item rewards. To this end, we adopt the Boltzmann-rational model for pairwise comparisons, which extends the Bradley-Terry-Luce model by incorporating worker competencies. We derive an EM-based algorithm for learning under this model by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified $Q$ function in the E-step of the algorithm. This technique allows us to reduce our formulation to a matrix sensing problem, using which we establish theoretical convergence guarantees for our algorithm. We conduct extensive experiments on real-world and synthetic datasets. These experiments demonstrate the advantages of using our algorithm over several baselines and confirm its strong robustness to both spammers and adversarial workers, highlighting its practical effectiveness in realistic crowdsourcing and reward learning settings. The code and data is publicly available at https://github.com/KaustubhShejole/BoRa_EM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。