无需种子数据，自动生成高质量泰语指令数据集。
Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai
- 用大模型生成多样话题并检索维基内容，自动构造指令。
- 仅用5000条指令即达顶尖泰语模型效果。
- 适合低资源语言研究者快速构建训练数据。
我们提出一种面向低资源语言（以泰语为例）的指令微调大语言模型的数据高效合成方法。识别出有效指令数据集的三个关键属性：流畅性、多样性与文化背景。设计了一种无种子数据的合成框架，利用大模型生成多样化主题，从维基百科检索相关上下文，并创建问答、摘要、对话等任务的指令。实验表明，融合全部三项属性的最佳合成数据集，仅用5000条指令即可达到与使用数十万条指令训练的先进泰语模型相当的性能。代码与数据集已公开于 https://github.com/parinzee/seed-free-synthetic-instruct。
原文摘要 · Abstract (English)
We present a synthetic data approach for instruction-tuning large language models (LLMs) for low-resource languages in a data-efficient manner, specifically focusing on Thai. We identify three key properties that contribute to the effectiveness of instruction-tuning datasets: fluency, diversity, and cultural context. We propose a seed-data-free framework for generating synthetic instruction-tuning data that incorporates these essential properties. Our framework employs an LLM to generate diverse topics, retrieve relevant contexts from Wikipedia, and create instructions for various tasks, such as question answering, summarization, and conversation. The experimental results show that our best-performing synthetic dataset, which incorporates all three key properties, achieves competitive performance using only 5,000 instructions when compared to state-of-the-art Thai LLMs trained on hundreds of thousands of instructions. Our code and dataset are publicly available at https://github.com/parinzee/seed-free-synthetic-instruct.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。