用随机Frank-Wolfe算法高效学习人类偏好，少次比较也能精准排序大量选项。
Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe
- 基于随机Frank-Wolfe设计最优对比采样点，减少人工反馈次数
- 在仅需$K$选$1$的对比下，实现对$N$个选项的高精度偏好建模
- 适合需要少样本、高效率偏好学习的RLHF等任务
我们研究从有限比较反馈中学习人类偏好，该任务在机器学习中极为普遍，尤其在基于人类反馈的强化学习中具有变革性。问题被建模为从$K$选$K$的比较反馈中学习一个普莱克特-卢瑟模型，通常$K \ll N$。我们提出针对普莱克特-卢瑟目标的D-最优设计，定义了一个数据采集策略，仅需从所有${N \choose K}$个可行子集中选取最优的一小部分进行对比。主要算法挑战在于，即使快速的D-最优设计方法也需$O({N \choose K})$时间复杂度。为此，我们提出一种随机化弗兰克-沃尔夫（FW）算法，在每次迭代中对随机选择的变量求解线性最大化子问题。我们分析了该算法，并在合成数据和开源NLP数据集上进行了实证评估。
原文摘要 · Abstract (English)
We study learning of human preferences from a limited comparison feedback. This task is ubiquitous in machine learning. Its applications such as reinforcement learning from human feedback, have been transformational. We formulate this problem as learning a Plackett-Luce model over a universe of $N$ choices from $K$-way comparison feedback, where typically $K \ll N$. Our solution is the D-optimal design for the Plackett-Luce objective. The design defines a data logging policy that elicits comparison feedback for a small collection of optimally chosen points from all ${N \choose K}$ feasible subsets. The main algorithmic challenge in this work is that even fast methods for solving D-optimal designs would have $O({N \choose K})$ time complexity. To address this issue, we propose a randomized Frank-Wolfe (FW) algorithm that solves the linear maximization sub-problems in the FW method on randomly chosen variables. We analyze the algorithm, and evaluate it empirically on synthetic and open-source NLP datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。