小模型微调易翻车，高效方法比全量微调更稳。
The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning
- 用参数高效方法（如LoRA）替代全量微调，防止性能下降。
- 300M以下模型全量微调后准确率反而低于零样本基线。
- 135M模型用简单提示学习就胜过全量微调，适合边缘部署。
在边缘设备上部署小型语言模型（SLMs）需高效微调策略，以适应新任务而不损害通用能力。本研究对五个子10亿参数模型（135M-1B）在数学推理任务上进行基准测试，发现关键问题：全量微调（Full FT）会显著损害300M以下模型的性能，准确率常低于零样本基线，形成‘负迁移’。这表明参数高效微调（PEFT）不仅是效率选择，更是稳定性必需。研究发现，低秩适应（LoRA）与权重分解LoRA（DoRA）表现相近，但任务依赖性强：DoRA在复杂推理（GSM8K）中更优，而LoRA在模式匹配（OrcaMath）中占优。尤其在对齐模型（Qwen2.5-0.5B）上，全量微调被LoRA超越；在最小模型（SmolLM2-135M）上，甚至不如5次上下文学习。据此建议：所有对齐的子10亿模型默认采用PEFT，避免在500M以下架构使用全量微调，以防灾难性遗忘。实验可复现于https://github.com/gulguluu/tiny-slm-finetune-compare。
原文摘要 · Abstract (English)
Deploying Small Language Models (SLMs) on edge devices requires efficient fine-tuning strategies that adapt models to new tasks without degrading their general capabilities. In this study, we benchmark five sub-1B models (135M-1B) on mathematical reasoning tasks and uncover a critical vulnerability: Full Fine-Tuning (Full FT) actively harms performance in models under 300M parameters, often dropping accuracy below zero-shot baselines. This "negative transfer" makes Parameter-Efficient Fine-Tuning (PEFT) not just an efficiency preference, but a stability requirement. We find that while Low-Rank Adaptation (LoRA) and Weight-Decomposed LoRA (DoRA) perform comparably, their strengths vary by task; DoRA excels in complex reasoning (GSM8K), while LoRA dominates pattern matching (OrcaMath). In particular, Full FT is outperformed by LoRA on aligned models (Qwen2.5-0.5B) and even by simple 5-shot In-Context Learning on the smallest architectures (SmolLM2-135M). Based on these findings, we recommend defaulting to PEFT for all aligned sub-1B models and caution against Full FT for any architecture smaller than 500M parameters to prevent catastrophic forgetting. Reproduction of this work can be found at https://github.com/gulguluu/tiny-slm-finetune-compare.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。