arXiv:2410.14208cs.CLcs.AI2024-10ICLR被引 12

让教师模型生成更符合学生学习需求的训练数据,提升学习效果。

Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning

  • 用学生学习偏好指导教师生成数据,提高数据影响力。
  • 在Alpaca Eval和MT-Bench上分别提升18.35%和46.24%。
  • 比更强的GPT-4o生成的数据还优,适合教学型模型训练。

合成数据被广泛用于训练大语言模型,但其生成特性会引入噪声、无效及误导性学习信号。本文提出蒙特梭利指令(Montessori-Instruct),一种新型数据合成框架,将教师模型的数据生成能力针对学生模型的学习过程进行定制。具体地,利用合成数据点对学生学习的局部影响来刻画学生的偏好;再通过直接偏好优化(DPO)训练教师模型,生成符合学生偏好、更具影响力的合成数据。在Llama3-8B-Instruct(教师)与Llama3-8B(学生)上,于Alpaca Eval与MT-Bench上的实验表明,该方法相比标准合成方法分别相对提升18.35%和46.24%,甚至优于使用更强教师模型GPT-4o生成的数据。进一步分析验证了教师学习生成高影响力数据对学生成绩的促进作用、局部影响测量偏好的准确性,以及方法在不同学生模型上的鲁棒性。代码与数据已开源。

原文摘要 · Abstract (English)

Synthetic data has been widely used to train large language models, but their generative nature inevitably introduces noisy, non-informative, and misleading learning signals. In this paper, we propose Montessori-Instruct, a novel data synthesis framework that tailors the data synthesis ability of the teacher language model toward the student language model's learning process. Specifically, we utilize local data influence of synthetic training data points on students to characterize students' learning preferences. Then, we train the teacher model with Direct Preference Optimization (DPO) to generate synthetic data tailored toward student learning preferences. Experiments with Llama3-8B-Instruct (teacher) and Llama3-8B (student) on Alpaca Eval and MT-Bench demonstrate that Montessori-Instruct significantly outperforms standard synthesis methods by 18.35\% and 46.24\% relatively. Our method also beats data synthesized by a stronger teacher model, GPT-4o. Further analysis confirms the benefits of teacher's learning to generate more influential training data in the student's improved learning, the advantages of local data influence in accurately measuring student preferences, and the robustness of Montessori-Instruct across different student models. Our code and data are open-sourced at https://github.com/cxcscmu/Montessori-Instruct.

数据合成教学增强偏好优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。