预训练模型重用效果好坏，关键看任务与特征相关性。
An empirical study of task and feature correlations in the reuse of pre-trained models
- 通过实验分析任务和特征相关性对模型重用的影响。
- 任务相关性越强，重用效果越好，甚至低相关时仍能优于随机性能。
- 低相关任务应只重用底层网络，可反推任务相关性程度。
预训练神经网络在机器学习中被广泛重用。例如，Alice为特定任务训练模型，Bob将其部分网络用于不同任务，常取得良好效果。这种成功可归因于什么？本文提出一种实验框架，用于在仿真中研究影响重用效果的因素。结果表明，Bob的成功可能只是运气：其任务准确率随与Alice任务的相关性单调上升。即使任务与输入特征无明显相关性，由于Alice的网络和优化器选择，Bob仍可实现显著优于随机的性能。当任务间相关性低时，仅重用底层预训练层更优；我们据此假设，最优重训层数可反映任务与特征的相关性。最后，在受控真实场景中，若任务间存在语义相关性，Bob可有效重用Alice的预训练网络。
原文摘要 · Abstract (English)
Pre-trained neural networks are commonly used and reused in the machine learning community. Alice trains a model for a particular task, and a part of her neural network is reused by Bob for a different task, often to great effect. To what can we ascribe Bob's success? This paper introduces an experimental setup through which factors contributing to Bob's empirical success could be studied in silico. As a result, we demonstrate that Bob might just be lucky: his task accuracy increases monotonically with the correlation between his task and Alice's. Even when Bob has provably uncorrelated tasks and input features from Alice's pre-trained network, he can achieve significantly better than random performance due to Alice's choice of network and optimizer. When there is little correlation between tasks, only reusing lower pre-trained layers is preferable, and we hypothesize the converse: that the optimal number of retrained layers is indicative of task and feature correlation. Finally, we show in controlled real-world scenarios that Bob can effectively reuse Alice's pre-trained network if there are semantic correlations between his and Alice's task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。