用自生成指令提升大模型对齐效果，成本降36%、准确率升5.2%
SeDi-Instruct: Enhancing Alignment of Language Models through Self-Directed Instruction Generation
- 通过多样性筛选与迭代反馈，自动生成高质量指令
- 相比传统方法，准确率提升5.2%，数据生成成本降低36%
- 适合需要低成本高质指令数据的工业级大模型训练
大型语言模型（LLM）的快速发展推动了各类AI服务的落地。指令微调在将基础模型适配到目标领域以提供高质量服务方面至关重要。其核心挑战在于获取高质量的指令数据。Self-Instruct利用ChatGPT API自动生成指令数据，缓解了数据稀缺问题，但因大量丢弃低质量指令而效率低下，导致成本过高。为此，我们提出一种新型数据生成框架——自指导指令生成（SeDi-Instruct），结合基于多样性的过滤和迭代反馈任务生成机制。多样性过滤在不显著降低模型精度的前提下，通过增强批次内指令多样性减少对低质量指令的过度剔除，从而降低合成成本。迭代反馈任务生成将指令生成与训练任务融合，利用训练过程中的信息生成高质量指令集。实验表明，与传统方法相比，SeDi-Instruct使模型准确率提升5.2%，同时将数据生成成本降低36%。
原文摘要 · Abstract (English)
The rapid evolution of Large Language Models (LLMs) has enabled the industry to develop various AI-based services. Instruction tuning is considered essential in adapting foundation models for target domains to provide high-quality services to customers. A key challenge in instruction tuning is obtaining high-quality instruction data. Self-Instruct, which automatically generates instruction data using ChatGPT APIs, alleviates the data scarcity problem. To improve the quality of instruction data, Self-Instruct discards many of the instructions generated from ChatGPT, even though it is inefficient in terms of cost owing to many useless API calls. To generate high-quality instruction data at a low cost, we propose a novel data generation framework, Self-Direct Instruction generation (SeDi-Instruct), which employs diversity-based filtering and iterative feedback task generation. Diversity-based filtering maintains model accuracy without excessively discarding low-quality generated instructions by enhancing the diversity of instructions in a batch. This reduces the cost of synthesizing instruction data. The iterative feedback task generation integrates instruction generation and training tasks and utilizes information obtained during the training to create high-quality instruction sets. Our results show that SeDi-Instruct enhances the accuracy of AI models by 5.2%, compared with traditional methods, while reducing data generation costs by 36%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。