arXiv:2602.11215cs.LG2026-02被引 1

首次揭示科学多领域微调的四大规律,指导高效训练通用大模型。

Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learning

  • 构建五领域语料,对比全量微调与低秩方法学习模式。
  • 发现多领域训练波动大,提出平衡-多样性等四条实证规律。
  • 适合想提升模型跨学科泛化能力的研究者参考。

尽管大语言模型在单一科学领域微调中表现优异,但其在多学科情境下的学习动态仍不明确,而跨领域知识协同有望提升模型泛化能力与适用性。本文首次系统研究多学科微调,构建了涵盖五个科学领域的语料库,分析了全量微调、LoRA、LoRA-MoE及LoRA组合的训练模式。研究表明,多学科学习的性能波动显著高于单领域训练,并总结出四条稳定的经验规律:(1) 平衡-多样性:低资源学科需通过多样性感知采样缓解性能下降;(2) 合并-对齐:恢复指令遵循能力是实现跨领域协同的关键;(3) 优化-再扩展:参数规模扩大在缺乏前期设计优化时增益有限;(4) 共享-专业化:非对称的LoRA-MoE通过共享低秩投影,在极小可训练参数下实现稳健提升。这些规律构成了有原则的多学科微调实用指南,为开发具备通用性的科学大模型提供具体路径。

原文摘要 · Abstract (English)

While large language models (LLMs) have achieved strong performance through fine-tuning within individual scientific domains, their learning dynamics in multi-disciplinary contexts remains poorly understood, despite the promise of improved generalization and broader applicability through cross-domain knowledge synergy. In this work, we present the first systematic study of multi-disciplinary LLM fine-tuning, constructing a five-discipline corpus and analyzing learning patterns of full fine-tuning, LoRA, LoRA-MoE, and LoRA compositions. Particularly, our study shows that multi-disciplinary learning is substantially more variable than single-discipline training and distills four consistent empirical laws: (1) Balance-then-Diversity: low-resource disciplines degrade performance unless mitigated via diversity-aware upsampling; (2) Merge-then-Align: restoring instruction-following ability is critical for cross-discipline synergy; (3) Optimize-then-Scale: parameter scaling offers limited gains without prior design optimization; and (4) Share-then-Specialize: asymmetric LoRA-MoE yields robust gains with minimal trainable parameters via shared low-rank projection. Together, these laws form a practical recipe for principled multi-discipline fine-tuning and provide actionable guidance for developing generalizable scientific LLMs.

多领域学习LLM微调经验规律科学建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。