用少量示例自动生成高质量专用数据集，省时省力。
CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation
- 基于用户示例检索网络文档，再用大模型生成任务适配样本。
- 医学、生物等4类任务中，生成数据训练模型表现超人工标注数据46分。
- 对初始示例质量不敏感，适合无领域专家的快速数据构建场景。
构建特定任务的高质量数据集通常耗时且需专业知识。本文提出CRAFT方法，仅需少量用户撰写的示范样本，即可生成合成数据集。该方法利用大规模公开网络语料和基于相似性的文档检索，找出相关人类撰写文本，并通过指令微调的大语言模型将其转换为符合任务格式的样本，用于模型微调。实验表明，CRAFT能高效生成生物学、医学、常识问答（QA）及摘要任务的规模化数据集。基于CRAFT生成数据训练的模型在问答任务上表现优于或相当一般大模型，而在摘要任务上比使用人工标注数据训练的模型高出46偏好点。CRAFT优于Self- and Evol-Instruct等合成数据生成方法，且在初始示例质量波动时仍保持稳健。
原文摘要 · Abstract (English)
Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge. We propose Corpus Retrieval and Augmentation for Fine-Tuning (CRAFT), a method for generating synthetic datasets, given a small number of user-written few-shots that demonstrate the task to be performed. Given these examples, CRAFT uses large-scale public web-crawled corpora and similarity-based document retrieval to find other relevant human-written documents. Lastly, instruction-tuned large language models (LLMs) augment the retrieved documents into custom-formatted task samples, which then can be used for fine-tuning. We demonstrate that CRAFT can efficiently generate large-scale task-specific training datasets for four diverse tasks: biology, medicine, and commonsense question-answering (QA), as well as summarization. Our experiments show that CRAFT-based models outperform or match general LLMs on QA tasks, while exceeding models trained on human-curated summarization data by 46 preference points. CRAFT outperforms other synthetic dataset generation methods such as Self- and Evol-Instruct, and remains robust even when the quality of the initial few-shots varies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。