用'愉悦度'筛选梯度,只对有价值样本计算反向传播,提升训练效率。
Does This Gradient Spark Joy?
- 引入'愉悦度'(优势×惊喜度)作为前向信号,判断样本是否值得反向传播。
- 在MNIST和Transformer任务中跳过90%以上反向传播,学习效果几乎无损。
- 适合高成本训练场景,如大模型预训练或资源受限环境。
策略梯度对每个样本都执行反向传播,尽管代价高昂且多数样本学习价值低。本文提出愉悦度策略梯度(DG),通过前向传播计算学习价值:愉悦度 = 优势 × 惊喜度(负对数概率)。引入Kondo门机制,将愉悦度与计算成本比较,仅在样本值得时才进行反向传播,从而逼近质量-成本的帕累托前沿。在强化学习老虎机任务中,零成本门控可保留有效梯度信号并去除垂直噪声;愉悦度比价值与惊喜度的线性组合更具可靠性。在MNIST和Transformer词元反转任务中,Kondo门跳过绝大多数反向传播,同时保持接近全部的训练质量,且随着问题复杂度上升、反向传播开销增加,性能增益更显著。由于该机制容忍愉悦度近似值,廉价前向筛查可在昂贵反向传播前完成,提示了一种面向训练的推测性解码新范式。
原文摘要 · Abstract (English)
Policy gradient computes a backward pass for every sample, even though the backward pass is expensive and most samples carry little learning value. The Delightful Policy Gradient (DG) provides a forward-pass signal of learning value: \emph{delight}, the product of advantage and surprisal (negative log-probability). We introduce the \emph{Kondo gate}, which compares delight against a compute price and pays for a backward pass only when the sample is worth it, thereby tracing a quality--cost Pareto frontier. In bandits, zero-price gating preserves useful gradient signal while removing perpendicular noise, and delight is a more reliable screening signal than additive combinations of value and surprise. On MNIST and transformer token reversal, the Kondo gate skips most backward passes while retaining nearly all of DG's learning quality, with gains that grow as problems get harder and backward passes become more expensive. Because the gate tolerates approximate delight, a cheap forward pass can screen samples before expensive backpropagation, suggesting a speculative-decoding-for-training paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。