用ChatGPT生成推理数据，让小模型在自然语言推理上表现更好。
Enhancing SLM via ChatGPT and Dataset Augmentation
- 用ChatGPT生成两种推理数据，扩充ANLI数据集。
- 小模型在ANLI上准确率提升1.3%和2.3%。
- 无需人工标注，适合资源有限的NLP研究者。
本文探索通过ChatGPT-3.5-Turbo对自然语言推理（NLI）领域的小语言模型（SLM）进行策略性数据集增强。采用基于知识蒸馏的技术与合成数据增强方法，旨在缩小大语言模型（LLMs）与小语言模型（SLMs）之间的性能差距，同时避免高昂的人工标注成本。研究通过信息提取与有根据推理两种方式生成推理过程，丰富ANLI数据集，并在此基础上微调T5-Small模型，评估其在基准测试中的表现。结果表明，引入合成推理过程显著提升了模型对自然语言的理解能力，在ANLI数据集上分别实现1.3%和2.3%的分类准确率提升，证明了利用大模型进行数据增强的潜力。该方法不仅增强了小模型在复杂任务上的表现，还提供了一种低成本的微调路径。本工作推动了知识蒸馏与微调策略的理解，有助于构建更高效、更具能力的自然语言处理系统。
原文摘要 · Abstract (English)
This paper explores the enhancement of small language models through strategic dataset augmentation via ChatGPT-3.5-Turbo, in the domain of Natural Language Inference (NLI). By employing knowledge distillation-based techniques and synthetic dataset augmentation, we aim to bridge the performance gap between large language models (LLMs) and small language models (SLMs) without the immense cost of human annotation. Our methods involve two forms of rationale generation--information extraction and informed reasoning--to enrich the ANLI dataset. We then fine-tune T5-Small on these augmented datasets, evaluating its performance against an established benchmark. Our findings reveal that the incorporation of synthetic rationales significantly improves the model's ability to comprehend natural language, leading to 1.3\% and 2.3\% higher classification accuracy, respectively, on the ANLI dataset, demonstrating the potential of leveraging LLMs for dataset augmentation. This approach not only enhances the performance of smaller models on complex tasks but also introduces a cost-effective method for fine-tuning smaller language models. By advancing our understanding of knowledge distillation and fine-tuning strategies, this work contributes to the ongoing effort to create more capable and efficient NLP systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。