用主动学习减少标注数据量，高效生成高质量偏好数据
ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning
- 基于不确定性估计动态选择最需标注的模型回复对
- 仅用1/6标注数据即达传统方法同等甚至更好效果
- 适合资源有限或专家领域，可直接用于大模型对齐
强化学习从人类反馈（RLHF）已成为对齐大语言模型的标准方法，但其效果受限于偏好数据获取成本高，尤其在低资源和专家领域。为此，我们提出ACTIVEULTRAFEEDBACK，一个模块化主动学习流程，利用不确定性估计动态识别最具信息量的回复进行标注。该流程系统评估了标准响应选择方法，并引入两种新方法：双重逆向汤普森采样（DRTS）与DELTAUCB，二者优先选择预测质量差距大的回复对，基于最新研究显示此类配对能提供良好微调信号。实验表明，ACTIVEULTRAFEEDBACK生成的高质量数据集显著提升下游性能，尤其在仅使用静态基线1/6标注数据时，仍能达到相当或更优结果。代码与数据集已开源。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。