用知识蒸馏压缩大模型，让小模型在问答任务上仍保持高效率。
Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
- 通过知识蒸馏从大模型学习，生成参数更少的小模型。
- 小模型参数减少57.1%，性能保留超90%。
- 少量提示（one-shot）比零样本更有效，适合资源受限场景。
大型语言模型（LLMs）在多种自然语言处理任务中表现卓越，但其计算开销限制了在资源受限环境中的部署。本文研究了知识蒸馏（KD）在问答（QA）任务中对LLMs的压缩极限。我们在SQuAD和MLQA两个问答基准上，评估了从Pythia和Qwen2.5系列模型蒸馏出的学生模型在零样本和单样本提示条件下的表现。结果表明，学生模型在参数量最多减少57.1%的情况下，仍能保持超过90%的教师模型性能。此外，单样本提示相比零样本设置带来额外性能提升。这些发现凸显了模型效率与任务性能之间的权衡，表明结合少量提示的知识蒸馏可生成高效且具备能力的问答系统，适用于资源受限的应用场景。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated outstanding performance across a range of NLP tasks, however, their computational demands hinder their deployment in real-world, resource-constrained environments. This work investigates the extent to which LLMs can be compressed using Knowledge Distillation (KD) while maintaining strong performance on Question Answering (QA) tasks. We evaluate student models distilled from the Pythia and Qwen2.5 families on two QA benchmarks, SQuAD and MLQA, under zero-shot and one-shot prompting conditions. Results show that student models retain over 90% of their teacher models' performance while reducing parameter counts by up to 57.1%. Furthermore, one-shot prompting yields additional performance gains over zero-shot setups for both model families. These findings underscore the trade-off between model efficiency and task performance, demonstrating that KD, combined with minimal prompting, can yield compact yet capable QA systems suitable for resource-constrained applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。