用小模型高效筛选大模型训练数据，省资源还提效果
Small Language Model as Data Prospector for Large Language Model
- 用小语言模型代替大模型筛选优质指令数据
- 效率提升58倍，性能仅降1-2%
- 适合资源有限但需高质量数据的LLM训练场景
指令数据的质量直接影响微调后大语言模型的性能。此前，Li et al. (2023) 提出 NUGGETS，通过识别单样本学习后能显著提升不同任务性能的指令实例，从大规模数据集中筛选高质量数据。本文提出 SuperNUGGETS，是 NUGGETS 的改进版本，专为效率与性能优化。SuperNUGGETS 采用小语言模型（SLM）替代大语言模型（LLM）进行数据过滤，并优化预定义测试集。实验结果表明，SuperNUGGETS 性能相比 NUGGETS 仅下降 1-2%，但效率提升达 58 倍。相较于原始 NUGGETS，SuperNUGGETS 因资源消耗显著降低，具备更高的实用价值。
原文摘要 · Abstract (English)
The quality of instruction data directly affects the performance of fine-tuned Large Language Models (LLMs). Previously, \cite{li2023one} proposed \texttt{NUGGETS}, which identifies and selects high-quality quality data from a large dataset by identifying those individual instruction examples that can significantly improve the performance of different tasks after being learnt as one-shot instances. In this work, we propose \texttt{SuperNUGGETS}, an improved variant of \texttt{NUGGETS} optimised for efficiency and performance. Our \texttt{SuperNUGGETS} uses a small language model (SLM) instead of a large language model (LLM) to filter the data for outstanding one-shot instances and refines the predefined set of tests. The experimental results show that the performance of \texttt{SuperNUGGETS} only decreases by 1-2% compared to \texttt{NUGGETS}, but the efficiency can be increased by a factor of 58. Compared to the original \texttt{NUGGETS}, our \texttt{SuperNUGGETS} has a higher utility value due to the significantly lower resource consumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。