分析大模型跨任务迁移中的隐藏因素，揭示性能提升的真正原因。
Latent Traits and Cross-Task Transfer: Deconstructing Dataset Interactions in LLM Fine-tuning
- 构建迁移矩阵与降维分析框架，挖掘模型潜在能力
- 发现性能提升受数据分布和语言特征影响大于表面相似性
- 适合关注模型迁移规律与训练数据设计的研究者
大型语言模型被广泛应用于各种未在训练中出现过的任务。由于难以为所有任务收集高质量数据，通常依赖跨任务迁移学习并应对分布外请求。为此，我们提出一个分析框架，通过构建迁移学习矩阵与维度压缩，剖析跨任务交互。我们训练并分析了10个模型，识别出隐含能力（如推理、情感分类、自然语言理解、算术）并发现迁移的副作用。结果表明，性能提升常无法用数据集表面相似性或源数据质量解释，而更受源数据隐藏统计特性（如类别分布、生成长度倾向）及特定语言特征影响。该研究揭示了迁移学习的复杂动态，为更可预测、高效的LLM适配提供思路。
原文摘要 · Abstract (English)
Large language models are increasingly deployed across diverse applications. This often includes tasks LLMs have not encountered during training. This implies that enumerating and obtaining the high-quality training data for all tasks is infeasible. Thus, we often need to rely on transfer learning using datasets with different characteristics, and anticipate out-of-distribution requests. Motivated by this practical need, we propose an analysis framework, building a transfer learning matrix and dimensionality reduction, to dissect these cross-task interactions. We train and analyze 10 models to identify latent abilities (e.g., Reasoning, Sentiment Classification, NLU, Arithmetic) and discover the side effects of the transfer learning. Our findings reveal that performance improvements often defy explanations based on surface-level dataset similarity or source data quality. Instead, hidden statistical factors of the source dataset, such as class distribution and generation length proclivities, alongside specific linguistic features, are actually more influential. This work offers insights into the complex dynamics of transfer learning, paving the way for more predictable and effective LLM adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。