通过提前启动反向传播，提升推荐系统训练速度。
ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems
- 将前向传递后立即启动反向传播，利用空闲计算资源。
- 在大规模点击率模型上提升9.51%训练吞吐量。
- 适合追求训练效率的推荐系统研发人员。
轻量级代理模型可实现快速实验，但其小规模内核常导致现代加速器利用率不足。传统训练将前向与反向传播分阶段执行，使两者无法共享闲置算力。本文将前向-前向（FF）机制中的解耦思想重新诠释为调度原语：给定局部目标，解耦子图输出可移除下游梯度依赖，使其反向传播在前向结束后即可立即启动。ERASE在独立CUDA流中提前发起每个解耦子图的反向传播，使其与后续前向计算重叠执行。在轻量级Transformer上的执行追踪显示，当某内核已饱和设备时，重叠将受限。在大规模点击率模型上，解耦六个密集子架构可将训练吞吐量提升最高达9.51%，同时保持归一化熵接近基线水平。
原文摘要 · Abstract (English)
Lightweight proxy models enable rapid experimentation without repeatedly training frontier-scale systems, but their small kernels often leave modern accelerators underutilized. Conventional training compounds this inefficiency by scheduling the forward and backward passes as disjoint phases, so spare capacity in one cannot be filled by work from the other. We reinterpret the detachment mechanism of Forward-Forward (FF) as a scheduling primitive: given a local objective, detaching a block's output removes downstream gradient dependencies, making its backward pass ready when its forward pass finishes. ERASE launches each detached subgraph's backward pass early on a separate CUDA stream, overlapping it with subsequent forward work. Execution trace on a lightweight transformer demonstrates this overlap and its limit: a kernel that saturates the device leaves no capacity for concurrency. On a large-scale click-through-rate model, detaching six dense subarchitectures improves training throughput by up to $9.51\%$ while keeping normalized entropy close to the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。