用轻量模型快速筛选适合迁移学习的中间任务。
Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning
- 构建嵌入空间映射（ESM）替代全量微调，加速任务选择。
- 在1.2万个任务对上验证，选优效率提升278倍，耗时仅1/10。
- 适合资源受限场景下高效选择最优迁移任务。
中间任务迁移学习可显著提升模型性能。例如，情感检测数据不足时，先在情感分类数据集上微调语言模型，能大幅改善效果。但如何选择合适的迁移任务？以往方法虽有效，却难以应用于大规模源任务池，因其需对所有源语言模型进行前向传播。本文提出嵌入空间映射（ESMs），一种轻量级神经网络，可近似微调语言模型的效果。我们开展了NLP任务可迁移性与任务选择的最大规模研究，覆盖12,000个源-目标任务对。实验表明，将ESM应用于原有方法，执行时间减少至1/10，磁盘占用降低278倍，同时保持高选择性能（平均后悔度@5得分为2.95）。
原文摘要 · Abstract (English)
Intermediate task transfer learning can greatly improve model performance. If, for example, one has little training data for emotion detection, first fine-tuning a language model on a sentiment classification dataset may improve performance strongly. But which task to choose for transfer learning? Prior methods producing useful task rankings are infeasible for large source pools, as they require forward passes through all source language models. We overcome this by introducing Embedding Space Maps (ESMs), light-weight neural networks that approximate the effect of fine-tuning a language model. We conduct the largest study on NLP task transferability and task selection with 12k source-target pairs. We find that applying ESMs on a prior method reduces execution time and disk space usage by factors of 10 and 278, respectively, while retaining high selection performance (avg. regret@5 score of 2.95).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。