用大模型生成题目和评分，让小模型学会高效答题。
LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering
- 用大模型自动生成多选题数据并打分
- 小模型在生成数据上训练，准确率提升10.4个百分点
- 适合资源有限却需高效少样本答题的场景
多选题问答（MCQA）在医疗、法律、教育等领域有广泛应用。构建高质量数据集成本高昂，因此少样本学习至关重要。尽管大语言模型（LLMs）可实现少样本学习，但其高计算开销限制了实际应用。本文提出一种简单有效的方法：利用LLMs生成包含问题与选项的MCQA数据，并为选项赋予概率评分；随后，基于这些生成数据与评分，使用知识蒸馏损失对更轻量的encoder-only模型DeBERTa-v3-base进行微调。在Massive Multitask Language Understanding（MMLU）基准上的实验表明，该方法将准确率从28.9%提升至39.3%，相较直接在5个示例上微调的基线提升了超过10%。结果证明，大模型驱动的数据生成与知识蒸馏在少样本MCQA中具有显著有效性。
原文摘要 · Abstract (English)
Multiple Choice Question Answering (MCQA) is an important problem with numerous real-world applications, such as medicine, law, and education. The high cost of building MCQA datasets makes few-shot learning pivotal in this domain. While Large Language Models (LLMs) can enable few-shot learning, their direct application in real-world scenarios is often hindered by their high computational cost. To address this challenge, we propose a simple yet effective approach that uses LLMs for data generation and scoring. Our approach utilizes LLMs to create MCQA data which contains questions and choices, and to assign probability scores to the generated choices. We then use the generated data and LLM-assigned scores to finetune a smaller and more efficient encoder-only model, DeBERTa-v3-base by leveraging distillation loss. Extensive experiments on the Massive Multitask Language Understanding (MMLU) benchmark demonstrate that our method improves accuracy from 28.9% to 39.3%, representing a gain of over 10% compared to a baseline finetuned directly on 5-shot examples. This shows the effectiveness of LLM-driven data generation and knowledge distillation for few-shot MCQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。