arXiv:2410.10210cs.CL2024-10

用高质量数据微调,少量计算即可让大模型生成更长内容。

Minimum Tuning to Unlock Long Output from LLMs with High Quality Data as the Key

  • 用精心筛选的长文本数据微调,提升模型输出长度能力。
  • 仅需少量数据和计算量,就能达到与全量训练相当的效果。
  • 方法通用性强,适用于多种主流大模型,适合实际部署。

随着大语言模型逐步支持更长上下文,其生成长文本的能力存在显著差异。研究表明,这种差距主要源于对齐训练阶段缺乏长输出数据。为此,本文探索了数据质量在微调模型长文本生成能力中的作用,并验证了从人类对齐模型(如指令或对话模型)出发进行微调的可行性。通过精心的数据筛选,我们仅用少量训练数据实例和计算资源,便实现了与全量训练相当的性能提升。此外,我们将该微调方法应用于多个不同模型,结果表明尽管各模型原始状态下长文本生成能力各异,但经过高质量数据微调后,所有模型均获得显著改进。相关数据集、代码及微调模型已开源,供公开使用。

原文摘要 · Abstract (English)

As large language models rapidly evolve to support longer context, there is a notable disparity in their capability to generate output at greater lengths. Recent study suggests that the primary cause for this imbalance may arise from the lack of data with long-output during alignment training. In light of this observation, attempts are made to re-align foundation models with data that fills the gap, which result in models capable of generating lengthy output when instructed. In this paper, we explore the impact of data-quality in tuning a model for long output, and the possibility of doing so from the starting points of human-aligned (instruct or chat) models. With careful data curation, we show that it possible to achieve similar performance improvement in our tuned models, with only a small fraction of training data instances and compute. In addition, we assess the generalizability of such approaches by applying our tuning-recipes to several models. our findings suggest that, while capacities for generating long output vary across different models out-of-the-box, our approach to tune them with high-quality data using lite compute, consistently yields notable improvement across all models we experimented on. We have made public our curated dataset for tuning long-writing capability, the implementations of model tuning and evaluation, as well as the fine-tuned models, all of which can be openly-accessed.

长文本生成数据质量轻量微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。