将上下文学习知识蒸馏到小模型，大幅压缩体积同时提升泛化性能。
In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning
- 用知识蒸馏把大模型的上下文学习能力迁移到125M小模型
- 模型大小从2.5GB减至0.25GB，出域准确率提升近50%
- 适合资源受限场景下的高效少样本微调
我们在OPT-1.3B模型上对自然语言推理任务应用少样本上下文学习，并采用知识蒸馏将上下文信息内化，使模型参数量从13亿降至1.25亿，模型体积从2.5GB缩减至0.25GB。相比同类规模模型仅使用上下文学习的方法，该上下文蒸馏方法在跨域任务上的准确率提升近50%，展现出优于提示学习方法的知识迁移能力。此外，该方法相比传统基于模式的微调，在内存消耗降低60%的同时，出域准确率提升20%。
原文摘要 · Abstract (English)
We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, reducing model parameter from 1.3B to 125M and achieving a size reduction from 2.5GB to 0.25GB. Compared to using in-context learning alone on similarly sized models, this context distillation approach achieved a nearly 50% improvement in out-of-domain accuracy, demonstrating superior knowledge transfer capabilities over prompt-based methods. Furthermore, this approach reduced memory consumption by up to 60% while delivering a 20% improvement in out-of-domain accuracy compared to conventional pattern-based fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。