发现预训练到微调的知识迁移存在显著差异，不同任务表现不一。
The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuning
- 通过相关性分析跨阶段能力迁移规律
- 准确率与置信度在不同模型规模下表现相反
- 为基准选择和数据优化提供实证指导
理解语言模型能力从预训练到监督微调（SFT）的迁移机制，是高效模型开发和数据管理的基础。本文围绕四个核心问题展开：预训练建立的准确率与置信度排序在微调后是否保持？哪些基准能作为跨阶段可靠预测指标？迁移动态如何随模型规模变化？模型置信度与准确率的对齐程度如何，且这种模式是否跨阶段延续？我们通过一系列相关性协议，在多种数据混合和模型规模下分析准确率与置信度指标。实验表明，知识迁移可靠性在不同能力类别、基准和规模间差异巨大，准确率与置信度表现出截然不同甚至对立的缩放特性。这些发现揭示了预训练决策与下游结果间的复杂互动，为基准选择、数据筛选和高效模型开发提供了可操作的指导。
原文摘要 · Abstract (English)
Understanding how language model capabilities transfer from pretraining to supervised fine-tuning (SFT) is fundamental to efficient model development and data curation. In this work, we investigate four core questions: RQ1. To what extent do accuracy and confidence rankings established during pretraining persist after SFT? RQ2. Which benchmarks serve as robust cross-stage predictors and which are unreliable? RQ3. How do transfer dynamics shift with model scale? RQ4. How well does model confidence align with accuracy, as a measure of calibration quality? Does this alignment pattern transfer across training stages? We address these questions through a suite of correlation protocols applied to accuracy and confidence metrics across diverse data mixtures and model scales. Our experiments reveal that transfer reliability varies dramatically across capability categories, benchmarks, and scales -- with accuracy and confidence exhibiting distinct, sometimes opposing, scaling dynamics. These findings shed light on the complex interplay between pretraining decisions and downstream outcomes, providing actionable guidance for benchmark selection, data curation, and efficient model development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。