无需反向传播，用输出误差直接调整模型，提速超2倍且内存降低40%。
Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation)
- 仅用输出层误差信号，不传梯度也不建计算图，实现无反向传播微调
- 在3个大模型上达到与全量微调相当的性能，推理指标波动小于种子噪声
- 两分钟诊断工具可判断哪些层适合这种快速适配，适合高效部署场景
前向传递仅需（FPO）训练大型语言模型时无需通过模型主体进行反向传播，相比标准微调实现2.7–3.2倍吞吐提升，峰值训练内存减少约40%，且离域基准测试结果与基线在种子噪声范围内一致，而全网络微调无法稳定复现此性质。FPO基于一个关键观察：在Transformer的深层，输出层预测误差与真实梯度的余弦相似度为0.47–0.59，覆盖六个公开模型。我们提出一个两分钟诊断工具，可量化任意模型各层的该近似程度，识别出适合后期层适配的区域。根据诊断结果，FPO在输出层计算一次误差信号，并将其应用于目标层。层间不传播信号，全程不构建自动求导图。我们在三种模型家族（OLMo-2-7B、Qwen3-8B、Falcon3-7B）上评估FPO，均实现领域内困惑度下降，且在MMLU、ARC-Challenge、HellaSwag和Winogrande上表现与基线在种子噪声范围内无显著差异。将SFT局部化至FPO目标层虽可行，但耗时达FPO的2.2倍。
原文摘要 · Abstract (English)
Forward-Pass-Only MLP training (FPO) adapts large language models without a backward pass through the model body, achieving 2.7--3.2x the throughput of standard fine-tuning at ~40% less peak training memory, while leaving off-domain benchmarks within seed-noise of baseline, a property that full-network fine-tuning does not reliably reproduce. FPO rests on a single empirical observation: at late layers of a transformer, the output-layer prediction error approximates the true gradient with cosine similarity 0.47--0.59 across six public models we survey. We introduce a two-minute diagnostic that quantifies this approximation per layer for any model, identifying where late-layer adaptation is viable. Informed by the diagnostic, FPO computes a single error signal at the output and applies it to each target layer. No signal is propagated between layers, and no autograd graph is constructed at any point. We evaluate FPO on three model families (OLMo-2-7B, Qwen3-8B, Falcon3-7B). Across all three, FPO produces in-domain perplexity improvement and leaves MMLU, ARC-Challenge, HellaSwag, and Winogrande within seed-noise of baseline. Localizing SFT to FPO's target layers to enter this regime is also feasible, but at 2.2x the wall-clock cost of FPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。