LoCA通过一次校准实现后续无需反向传播的高效模型微调，降低显存和计算开销。
LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment
- 一次反向传播校准各层梯度映射，后续仅用前向传播拟合低秩适配器
- 在5个任务上比LoRA平均降低26%-29%峰值显存，推理速度提升43%-48%
- 适合资源受限场景下的快速模型适配，支持多尺寸模型通用
参数高效微调虽减少可训练参数，但仍需对冻结主干网络重复进行端到端反向传播，每次适配步骤都依赖具备反向计算能力的硬件并需存储或重算激活值。本文提出局部信用分配（LoCA），一种两阶段小扰动适应方法。首先通过一次反向传播，在每个Transformer块中建立从最终预测误差到局部隐藏状态修正的低秩映射；随后利用这些映射，基于前向激活构造分块回归目标，并通过闭式岭回归求解低秩适配器。此后无需任何主干反向传播。我们在0.5B至14B的Qwen2.5模型上评估了五个判别性基准任务。在25项任务-规模对比中的16项中，LoCA的评估交叉熵低于对应LoRA结果。其完整运行峰值GPU内存比LoRA低26%-29%。校准后CPU稳定内存降低36%-52%，每步时间缩短43%-48%。一个共享的归一化候选集可跨所有测试的Qwen2.5尺寸及SmolLM2-1.7B复用。因此，LoCA将全局信用分配成本摊销为一次校准，实现在反向传播不可行时的前向仅微调。代码已公开。
原文摘要 · Abstract (English)
Parameter-efficient post-training reduces the number of trainable parameters, but still requires repeated end-to-end backpropagation through the frozen backbone. Every adaptation step therefore needs backward-capable hardware and must store or recompute activations. We ask whether this repeated backward chain can be replaced by a one-time calibration. We introduce Local Credit Assignment (LoCA), a two-stage method for small-shift adaptation. One probe backward pass fits a low-rank map at each transformer block from the final prediction error to a local hidden-state correction. LoCA then reuses these maps to form blockwise regression targets from forward activations and fits low-rank adapters with closed-form ridge solves. No further backbone backward pass is required. We evaluate LoCA on five discriminative benchmarks with Qwen2.5 models from 0.5B to 14B. In 16 of 25 reported task--scale comparisons, LoCA yields lower evaluation cross-entropy than the corresponding LoRA run. Its measured full-run GPU peak, including calibration, is 26--29\% lower than LoRA's. After calibration, its CPU steady-state memory is 36--52\% lower and its per-pass time is 43--48\% lower. A shared scale-normalized candidate set is reused across all tested Qwen2.5 sizes and on SmolLM2-1.7B. LoCA thus amortizes global credit assignment into one calibration and enables later forward-only tuning when repeated backpropagation is impractical. The code associated with this paper is available \href{https://github.com/Xia12121/LoCA}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。