根据任务匹配预训练目标，显著提升生成与问答模型性能。
Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models
- 提出MTO框架，自动匹配任务与预训练目标
- 少样本下性能提升超120%，全数据集也更优
- 指导提示调优，适合定制化任务的模型优化
提示学习已成为自然语言处理的主流范式。本研究探讨了不同预训练目标对编码器-解码器型预训练语言模型在生成与问答任务中的影响，重点关注常识知识检索与补全。研究发现，在预训练与微调阶段融合多目标可带来显著收益。为此提出匹配任务到目标（MTO）框架，实现基于识别目标的自动化数据准备与无监督适配。微调阶段设计与预训练目标对齐的新模板，当与任务需求一致时，少样本设置下性能较传统方法提升超120%。该策略在少样本场景中显著优于现有工作，即使在全数据集下仍超越基线。进一步扩展至提示调优，为更有效的软提示工程提供指导。所提方法显著提升提示调优表现，对特定任务的模型选择与优化具有重要价值。代码已开源：https://github.com/puraminy/MTO/
原文摘要 · Abstract (English)
Prompt-based learning has emerged as a dominant paradigm in natural language processing. This study explores the impact of diverse pre-training objectives on the performance of encoder-decoder pre-trained language models across generation and question answering tasks, with a focus on commonsense knowledge retrieval and completion. We highlight the benefits of incorporating multiple objectives during both pre-training and fine-tuning stages. We introduce the Match Task to Objective (MTO) framework and methods for determining the appropriate objective for a given task. This framework offers automated methods to prepare task-related data for adaptation through unsupervised training, based on the identified objective. In the fine-tuning stage, we design novel templates that align with the objectives of the pre-training and adaptation stages. When aligned with task requirements, these strategies can achieve a performance gain of over 120\% compared to conventional methods in few-shot settings. They significantly outperform related works in few-shot settings and exceed the baseline even in full-dataset scenarios. Furthermore, we extend this approach to include prompt-tuning methodologies, providing guidance for more effective soft prompt engineering and optimization. Our strategies significantly enhance prompt-tuning performance as well. These insights hold substantial value, precisely guiding the selection and optimization of models customized for specific tasks. Code is available at https://github.com/puraminy/MTO/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。