用元学习适配器，让大模型在少量数据下也能高效生成改写句。
Learning to Adapt to Low-Resource Paraphrase Generation
- 分三阶段训练：先无监督预训练，再源域元训练，最后目标域微调。
- 仅用1%目标数据和2%参数量，性能媲美全量训练模型。
- 适合数据稀缺场景，尤其适用于跨领域文本改写任务。
文本改写是自然语言处理中的经典任务，依赖大规模语料取得显著进展。然而，将模型迁移至新领域时，常因领域差异导致性能下降，尤其在数据稀疏时更为严重。同时，使用大型预训练语言模型（PLMs）在少量标注数据上训练易出现过拟合。为此，我们提出LAPA，一种基于元学习优化的PLM适配器。LAPA采用三阶段训练策略：1）在无监督语料上预训练PLMs；2）插入适配层并使用源域标注数据进行元训练；3）在少量目标域标注数据上微调适配器。该方法使模型先学习基础语言知识，再掌握改写任务，最后适应目标任务。实验表明，LAPA在三个基准数据集上均达到最优性能，在有监督、无监督及低资源设置下表现突出。仅需1%的目标域标注数据和2%的可训练参数，即可实现与以往方法相当的性能。
原文摘要 · Abstract (English)
Paraphrase generation is a longstanding NLP task and achieves great success with the aid of large corpora. However, transferring a paraphrasing model to another domain encounters the problem of domain shifting especially when the data is sparse. At the same time, widely using large pre-trained language models (PLMs) faces the overfitting problem when training on scarce labeled data. To mitigate these two issues, we propose, LAPA, an effective adapter for PLMs optimized by meta-learning. LAPA has three-stage training on three types of related resources to solve this problem: 1. pre-training PLMs on unsupervised corpora, 2. inserting an adapter layer and meta-training on source domain labeled data, and 3. fine-tuning adapters on a small amount of target domain labeled data. This method enables paraphrase generation models to learn basic language knowledge first, then learn the paraphrasing task itself later, and finally adapt to the target task. Our experimental results demonstrate that LAPA achieves state-of-the-art in supervised, unsupervised, and low-resource settings on three benchmark datasets. With only 2\% of trainable parameters and 1\% labeled data of the target task, our approach can achieve a competitive performance with previous work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。