LoRA效果差异主因是批大小，调好后简单版本也能胜复杂变体。
Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA
- 发现批大小是影响LoRA性能的关键变量，而非模型结构差异。
- 正确调参后，基础LoRA性能可媲美复杂变体。
- 提出低成本批大小调优方法，适合评估各类LoRA变体。
低秩适应（LoRA）是微调大语言模型的标准方法，但其众多变体在相同基准上报告出矛盾的实验结果。我们发现这些矛盾主要源于一个被忽视的因素：批大小。当批大小合理调整时，原始的LoRA通常能与更复杂的变体性能相当。我们进一步提出一种基于代理的、成本高效的批大小调优策略，揭示了秩、数据集规模和模型容量对最优批大小的影响。研究结果将批大小从次要实现细节提升为首要设计参数，弥合了先前的不一致，使LoRA变体的评估更加可靠。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) is a standard approach for fine-tuning large language models, yet its many variants report conflicting empirical gains, often on the same benchmarks. We show that these contradictions arise from a single overlooked factor: the batch size. When properly tuned, vanilla LoRA often matches the performance of more complex variants. We further propose a proxy-based, cost-efficient strategy for batch size tuning, revealing the impact of rank, dataset size, and model capacity on the optimal batch size. Our findings elevate batch size from a minor implementation detail to a first-order design parameter, reconciling prior inconsistencies and enabling more reliable evaluations of LoRA variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。