自动调整预训练目标权重，让模型更懂下游任务。
TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretraining
- 根据下游任务反馈自动分配不同预训练目标的权重。
- 在分子属性预测和自然语言理解任务上显著超越基线。
- 无需手动调参，减少计算开销，通用性强。
大规模通用领域预训练后进行下游特定微调已成为机器学习主流范式。然而，预训练与目标任务领域差异仍可能导致性能下降，凸显任务自适应持续预训练（TAP）的必要性。现有TAP方法通常在任务特定无标签数据上继续预训练，或引入额外无监督学习目标以增强模型能力。尽管许多TAP方法使用多个预训练目标进行持续预训练，但其目标间权衡参数常需人工设定，导致效果不佳且计算成本高。本文提出TapWeight框架，通过下游反馈自动确定每个预训练目标的最优重要性，通过求解多层级优化问题实现目标重加权。我们在分子属性预测和自然语言理解任务上应用TapWeight，结果显著优于基线方法。实验验证了其有效性和泛化能力。
原文摘要 · Abstract (English)
Large-scale general domain pretraining followed by downstream-specific finetuning has become a predominant paradigm in machine learning. However, discrepancies between the pretraining and target domains can still lead to performance degradation in certain cases, underscoring the need for task-adaptive continued pretraining (TAP). TAP methods typically involve continued pretraining on task-specific unlabeled datasets or introducing additional unsupervised learning objectives to enhance model capabilities. While many TAP methods perform continued pretraining with multiple pretraining objectives, they often determine the tradeoff parameters between objectives manually, resulting in suboptimal outcomes and higher computational costs. In this paper, we propose TapWeight, a task-adaptive pretraining framework which automatically determines the optimal importance of each pretraining objective based on downstream feedback. TapWeight reweights each pretraining objective by solving a multi-level optimization problem. We applied TapWeight to both molecular property prediction and natural language understanding tasks, significantly surpassing baseline methods. Experimental results validate the effectiveness and generalizability of TapWeight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。