提出可低成本预估自监督任务对目标性能影响的方法
Quantitative Estimation of Target Task Performance from Unsupervised Pretext Task in Semi/Self-Supervised Learning
- 基于假设的可学习性、可靠性和完备性构建理论框架
- 在上百个预训练任务上验证，预测值与真实性能高度相关
- 适合想快速筛选有效自监督任务的研究者使用
自监督学习中未标记数据的有效性依赖于特定场景下的合理假设，从而决定有益的无监督预训练任务选择。然而现有研究对SSL中的假设关注不足，导致预训练任务与目标场景的适配性只能在训练和验证后才能评估。本文聚焦无监督预训练任务背后的假设，探索低成本预先估计其对目标性能影响的可行性。通过严格推导，我们证明预训练任务对目标性能的影响取决于三个因素：模型层面的假设可学习性、数据层面的假设可靠性，以及目标层面的假设完备性。基于此理论，我们提出一种低成本量化估计方法，可准确预测实际目标性能。我们构建了一个包含百余个预训练任务的基准测试，实验表明，该方法的预测结果与大规模训练后的实际性能具有强相关性。
原文摘要 · Abstract (English)
The effectiveness of unlabeled data in Semi/Self-Supervised Learning (SSL) depends on appropriate assumptions for specific scenarios, thereby enabling the selection of beneficial unsupervised pretext tasks. However, existing research has paid limited attention to assumptions in SSL, resulting in practical situations where the compatibility between the unsupervised pretext tasks and the target scenarios can only be assessed after training and validation. This paper centers on the assumptions underlying unsupervised pretext tasks and explores the feasibility of preemptively estimating the impact of unsupervised pretext tasks at low cost. Through rigorous derivation, we show that the impact of unsupervised pretext tasks on target performance depends on three factors: assumption learnability with respect to the model, assumption reliability with respect to the data, and assumption completeness with respect to the target. Building on this theory, we propose a low-cost estimation method that can quantitatively estimate the actual target performance. We build a benchmark of over one hundred pretext tasks and demonstrate that estimated performance strongly correlates with the actual performance obtained through large-scale training and validation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。