用少量人工标注实现大模型对齐，效率提升近95%。
RLTHF: Targeted Human Feedback for LLM Alignment
- 结合大模型初筛与精准人工修正，动态识别难标注样本。
- 仅需6-7%人工标注量，达到全人工标注的对齐效果。
- 适合追求高效对齐的大模型训练团队使用。
由于强化学习中人类反馈（RLHF）高质量人工标注成本高昂，且人工智能反馈泛化能力有限，大语言模型对齐面临挑战。为此，我们提出一种人机协同框架RLTHF：先用大模型进行初步对齐，再通过奖励模型的奖励分布识别被错误标注的难样本，迭代地引入策略性人工修正，同时保留大模型正确标注的样本。在HH-RLHF和TL;DR数据集上的实验表明，RLTHF仅需6-7%的人工标注量即可达到全人工标注级别的对齐效果。此外，基于RLTHF筛选数据集训练的下游任务模型，性能优于使用全人工标注数据训练的模型，验证了该方法的有效性。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) to align with user preferences is challenging due to the high cost of quality human annotations in Reinforcement Learning from Human Feedback (RLHF) and the generalizability limitations of AI Feedback. To address these challenges, we propose RLTHF, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations to achieve full-human annotation alignment with minimal effort. RLTHF identifies hard-to-annotate samples mislabeled by LLMs using a reward model's reward distribution and iteratively enhances alignment by integrating strategic human corrections while leveraging LLM's correctly labeled samples. Evaluations on HH-RLHF and TL;DR datasets show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort. Furthermore, models trained on RLTHF's curated datasets for downstream tasks outperform those trained on fully human-annotated datasets, underscoring the effectiveness of RLTHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。