通过迭代训练和合并小模型，让低参数微调模型的表达能力持续提升。
BoostLoRA: Growing Effective Rank by Boosting Adapters
- 用梯度提升思路逐轮训练并合并极小适配器，逐步扩展有效秩
- 在Qwen2.5-3B上达89.1% GSM8K得分，超越全量微调和单次极小适配器
- 零推理开销，适合资源受限场景下的高效模型定制
参数高效微调（PEFT）方法在适配器规模与表达能力间存在权衡：极小参数适配器被限制在固定低秩子空间，即使延长训练也难突破性能上限。我们提出BoostLoRA，一种基于梯度提升的框架，通过在当前模型出错的样本上迭代训练并合并最小适配器来突破这一限制。采用ROTATE SVD基策略，每轮分配至正交子空间，使累积有效秩随轮数线性增长，而每轮适配器仍保持超低秩。合并后丢弃适配器，推理无额外开销。在Qwen2.5-3B上，BoostLoRA在GSM8K达到89.1%，在MATH-500达到68.8%，超过最佳单次极小参数适配器（TinyLoRA）和全量微调；代码生成任务中，在MBPP达57.2%，HumanEval达80.4%，而全量微调反而低于零样本基准。还验证了在ESM2-650M上的跨架构迁移与交叉熵训练。据我们所知，BoostLoRA是首个有效秩随训练增长的PEFT方法，实现每轮参数成本与总表征能力的解耦。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) methods face a tradeoff between adapter size and expressivity: ultra-low-parameter adapters are confined to fixed low-rank subspaces, capping performance even with extended training. We propose BoostLoRA, a gradient-boosting framework that overcomes this limit by iteratively training and merging minimal adapters on the examples the current model gets wrong. A ROTATE SVD basis strategy assigns each round to an orthogonal subspace, so cumulative effective rank grows linearly with the number of rounds while each adapter remains ultra-low-rank. After merging, adapters are discarded, leaving zero inference overhead. On Qwen2.5-3B, BoostLoRA reaches 89.1% on GSM8K and 68.8% on MATH-500, surpassing both the best single-shot ultra-low parameter adapter (TinyLoRA) and full fine-tuning; on code generation it reaches 57.2% on MBPP and 80.4% on HumanEval while full fine-tuning drops below the zero-shot baseline. We also demonstrate cross-architecture transfer on protein binding classification with ESM2-650M and cross-entropy training. BoostLoRA is, to our knowledge, the first PEFT method whose effective rank grows with training, separating per-round parameter cost from total representational capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。