用强化学习让小模型自动生成指令数据，省钱省力还更有效。
REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback
- 让小模型自己生成指令，减少人工标注
- 结合强化学习后，63%-66%任务表现更好
- 适合想低成本微调大模型的研究者
基于指令的大型语言模型在众多少样本或零样本自然语言处理任务中表现出色。然而，人工标注指令数据耗时、昂贵且数量有限、任务多样性不足。以往研究尝试通过半自动化、任务无关的方法从模型自身生成指令。许多方法依赖于GPT-3.5（175B）等大型API模型，成本高且查询受限。本文探索了三个开源小型模型（LLaMA 2-7B、LLaMA 2-13B、Mistral 7B）在半自动化框架下的表现，显著降低人工干预、投入与成本。进一步将强化学习训练算法引入该框架，实现性能提升。评估显示，相比先前方法，该基于强化学习的框架在63%-66%的任务中取得显著改进。
原文摘要 · Abstract (English)
Instruction-based Large Language Models (LLMs) have proven effective in numerous few-shot or zero-shot Natural Language Processing (NLP) tasks. However, creating human-annotated instruction data is time-consuming, expensive, and often limited in quantity and task diversity. Previous research endeavors have attempted to address this challenge by proposing frameworks capable of generating instructions in a semi-automated and task-agnostic manner directly from the model itself. Many of these efforts have relied on large API-only parameter-based models such as GPT-3.5 (175B), which are expensive, and subject to limits on a number of queries. This paper explores the performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B, and Mistral 7B, using a semi-automated framework, thereby reducing human intervention, effort, and cost required to generate an instruction dataset for fine-tuning LLMs. Furthermore, we demonstrate that incorporating a Reinforcement Learning (RL) based training algorithm into this LLMs-based framework leads to further enhancements. Our evaluation of the dataset reveals that these RL-based frameworks achieve a substantial improvements in 63-66% of the tasks compared to previous approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。