大规模实验揭示数据、层位与训练因素如何影响大模型对齐效果
Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
- 在1000多个受控模型上系统测试不同数据集与训练策略
- 困惑度能更好预测微调效果，中层权重变化最影响性能提升
- 结果表明需按模型定制策略，适合模型对齐研究者参考
监督微调（SFT）是使大语言模型与人类指令和价值观对齐的关键步骤，但其诸多方面仍不清晰。我们基于多种数据集（包括代码生成、数学推理和通用任务）在大量基础模型上进行了受控条件下的微调，共训练出1000多个SFT模型。通过分析发现，部分训练任务协同效应在所有模型中均存在，而其他则显著依赖模型特性，强调了模型特定策略的重要性。此外，我们证明困惑度能一致预测SFT有效性，常优于训练数据与评估基准间的表面相似性；中层权重变化与性能提升相关性最强。我们已公开这些1000多个SFT模型及基准结果，以加速后续研究。所有资源可在https://github.com/llm-jp/massive-sft获取。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is a critical step in aligning large language models (LLMs) with human instructions and values, yet many aspects of SFT remain poorly understood. We trained a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks, resulting in 1,000+ SFT models under controlled conditions. We then identified the dataset properties that matter most and examined the layer-wise modifications introduced by SFT. Our findings reveal that some training-task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies. Moreover, we demonstrate that perplexity consistently predicts SFT effectiveness, often surpassing superficial similarity between the training data and the benchmark, and that mid-layer weight changes correlate most strongly with performance gains. We release these 1,000+ SFT models and benchmark results to accelerate further research. All resources are available at https://github.com/llm-jp/massive-sft.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。