通过潜空间校准提升预训练模型微调效果
RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models
- 在编码器后引入校准模块，调整潜在表示以适配解码器
- 25个模型、8个任务上显著提升下游性能，4个基准任务优于主流方法
- 通用性强，可直接嵌入各类编码-解码架构模型
预训练语言模型（PLM）的微调已成为应用到下游任务的主流范式。然而，在有限微调条件下，模型编码器获得的表示与解码器最优输入之间仍存在差异。本文提出一种潜空间表示校准方法（RepCali），在编码器后的潜空间中集成特定校准模块，并将校准后的输出作为解码器输入。该方法适用于所有编码-解码架构的PLM，具有即插即用特性且实现简单。在25个基于PLM的模型、8个任务（涵盖英、中文数据集）上的大量实验表明，RepCali显著提升了各类PLM（包括大模型）的性能。在4个基准任务上的对比实验显示，其效果优于代表性微调基线方法。
原文摘要 · Abstract (English)
Fine-tuning pre-trained language models (PLMs) has become a dominant paradigm in applying PLMs to downstream tasks. However, with limited fine-tuning, PLMs still struggle with the discrepancies between the representation obtained from the PLMs' encoder and the optimal input to the PLMs' decoder. This paper tackles this challenge by learning to calibrate the representation of PLMs in the latent space. In the proposed representation calibration method (RepCali), we integrate a specific calibration block to the latent space after the encoder and use the calibrated output as the decoder input. The merits of the proposed RepCali include its universality to all PLMs with encoder-decoder architectures, its plug-and-play nature, and ease of implementation. Extensive experiments on 25 PLM-based models across 8 tasks (including both English and Chinese datasets) demonstrate that the proposed RepCali offers desirable enhancements to PLMs (including LLMs) and significantly improves the performance of downstream tasks. Comparison experiments across 4 benchmark tasks indicate that RepCali is superior to the representative fine-tuning baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。