用树搜索优化指令生成，让大模型自动生成更高质量的指令数据。
Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Search
- 基于蒙特卡洛树搜索，系统引导指令逐步优化
- 指令质量、多样性和复杂度平均分从2.19提升至3.81
- 在低资源场景下提升模型指令遵循准确率5%
指令微调是使语言模型与人类真实目标对齐的关键技术。高质量的指令数据对对齐效果至关重要，但人工构建成本高。现有研究尝试使用更强的LLM迭代优化指令数据，但缺乏方向控制，导致生成过程不确定性大、质量不稳定。本文提出通用可扩展框架IDEA-MCTS（基于蒙特卡洛树搜索的指令数据增强），通过树搜索与评估模型协同，高效引导每条指令向高质量方向演化，助力指令微调。实验表明，IDEA-MCTS显著提升种子指令数据质量，平均评价分数从2.19提升至3.81，涵盖质量、多样性与复杂度；在开放域基准测试中，该方法使大模型在低资源场景下的真实指令遵循能力平均提升5%。
原文摘要 · Abstract (English)
Instruction tuning is a crucial technique for aligning language models with humans' actual goals in the real world. Extensive research has highlighted the quality of instruction data is essential for the success of this alignment. However, creating high-quality data manually is labor-intensive and time-consuming, which leads researchers to explore using LLMs to synthesize data. Recent studies have focused on using a stronger LLM to iteratively enhance existing instruction data, showing promising results. Nevertheless, previous work often lacks control over the evolution direction, resulting in high uncertainty in the data synthesis process and low-quality instructions. In this paper, we introduce a general and scalable framework, IDEA-MCTS (Instruction Data Enhancement using Monte Carlo Tree Search), a scalable framework for efficiently synthesizing instructions. With tree search and evaluation models, it can efficiently guide each instruction to evolve into a high-quality form, aiding in instruction fine-tuning. Experimental results show that IDEA-MCTS significantly enhances the seed instruction data, raising the average evaluation scores of quality, diversity, and complexity from 2.19 to 3.81. Furthermore, in open-domain benchmarks, experimental results show that IDEA-MCTS improves the accuracy of real-world instruction-following skills in LLMs by an average of 5\% in low-resource settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。