用模板分类头提升小样本微调效率，让科研人员轻松用大模型。
PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning
- 将提示模板分类头与参数高效微调结合，实现轻量级小样本学习。
- 在三个基准数据集和一个真实研究数据集上表现媲美全量微调。
- 模块化设计易共享，适合资源有限的科研团队快速应用。
小样本学习与参数高效微调（PEFT）对于应对数据稀缺和语言模型规模不断增长的挑战至关重要，尤其在专业科学领域，研究人员常因缺乏专业知识和资源而难以将高性能语言模型适配到具体任务。本文提出PETapter，一种新方法，将PEFT与基于提示的模式分类头（PET-style）有效结合，在不带来显著计算开销的前提下，显著提升小样本学习能力。我们在三个标准NLP基准数据集及一个来自传播学研究的真实数据集上验证该方法。结果表明，PETapter不仅在性能上可媲美全量微调（使用模式挖掘训练，PET），还具有更高的可靠性、更强的参数效率，并支持更高程度的模块化与模块共享，使更多研究者能够便捷地在自身研究中使用高性能NLP方法。
原文摘要 · Abstract (English)
Few-shot learning and parameter-efficient fine-tuning (PEFT) are crucial to overcome the challenges of data scarcity and ever growing language model sizes. This applies in particular to specialized scientific domains, where researchers might lack expertise and resources to fine-tune high-performing language models to nuanced tasks. We propose PETapter, a novel method that effectively combines PEFT methods with PET-style classification heads to boost few-shot learning capabilities without the significant computational overhead typically associated with full model training. We validate our approach on three established NLP benchmark datasets and one real-world dataset from communication research. We show that PETapter not only achieves comparable performance to full few-shot fine-tuning using pattern-exploiting training (PET), but also provides greater reliability and higher parameter efficiency while enabling higher modularity and easy sharing of the trained modules, which enables more researchers to utilize high-performing NLP-methods in their research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。