智能选层让LoRA微调更快，不损失效果
Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures
- 用梯度探测识别关键层，只在相关层加LoRA
- 平均提速23.1%，最大达28%，且遗忘可控
- 适合大模型高效微调，尤其多架构场景
低秩适配（LoRA）已成为大语言模型参数高效微调的主流方法，但传统做法对所有Transformer层统一应用LoRA适配器，忽略其任务相关性。本文提出Aletheia，一种基于梯度引导的层选择方法，通过轻量级梯度探针识别最相关的层，并采用非对称秩分配仅在这些层上应用LoRA。在覆盖14种成功模型、8种架构家族（0.5B-72B参数，含密集与专家混合架构）的81组实验中，包括一次未成功的Pythia/GPT-NeoX尝试，Aletheia实现了15%-28%的训练速度提升（均值23.1%，p < 0.001），并保持了有限的额外遗忘，在MMLU、GSM8K和HumanEval基准集上表现基本一致。在所有测试架构与规模下，第一轮实验实现每模型100%提速，第二轮实验显示下游行为在可控退化范围内保持稳定。结果支持一个实用主张：智能层选择可显著提升LoRA微调效率，且不会造成显著下游性能损伤。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become the dominant parameter-efficient fine-tuning method for large language models, yet standard practice applies LoRA adapters uniformly to all transformer layers regardless of their relevance to the downstream task. We introduce Aletheia, a gradient-guided layer selection method that identifies the most task-relevant layers via a lightweight gradient probe and applies LoRA adapters only to those layers with asymmetric rank allocation. Across 81 experiment rows covering 14 successful models from 8 architecture families (0.5B-72B parameters, including dense and Mixture-of-Experts architectures), with one additional documented failed Pythia/GPT-NeoX attempt in Campaign 2, Aletheia achieves a 15-28% training speedup (mean 23.1%, p < 0.001) with bounded extra forgetting and broadly matched downstream behavior on the evaluated MMLU, GSM8K, and HumanEval benchmark pack. Across the tested families and scales, Campaign 1 shows a 100% per-model speed win rate and Campaign 2 shows broadly preserved downstream behavior within a bounded-degradation framing. Together these results support a practical model-economics claim: intelligent layer selection can make LoRA fine-tuning materially more efficient without introducing major downstream damage on the evaluated set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。