根据采样成本优化梯度下降，降低大模型训练的计算开销。
Cost-Aware Learning
- 按梯度大小和成本动态选择样本，实现低成本高效率优化。
- 在1.5B~8B模型上减少大量令牌消耗，性能不降反升。
- 适用于长序列强化学习，尤其适合语言模型的策略优化。
我们研究成本感知学习问题：在有限求和目标中，不同组件的采样成本各异。目标是在达到指定误差的前提下最小化总成本。本文提出成本感知SGD，基于梯度范数和成本设计采样分布。理论分析包括对基线算法的成本改进界、分布近似次优性的刻画以及下界证明。将理论洞察应用于语言模型的强化学习，发现优势幅度可作为梯度范数的高保真代理，并据此提出成本感知GRPO。在1.5B、4B和8B大语言模型上的实验表明，该算法显著减少策略优化中的令牌使用量，同时保持或超越基线准确率。
原文摘要 · Abstract (English)
We consider the problem of Cost-Aware Learning, where sampling different components of a finite-sum objective incurs different costs. The objective is to reach a target error while minimizing the total cost. We propose Cost-Aware SGD, which uses a distribution based on gradient norms and costs to sample components. We provide a thorough analysis of this algorithm, including cost-improvement bounds over baselines, a characterization of distribution proxy sub-optimality, and a lower bound. We apply our theoretical insights to reinforcement learning with language models, where the computational cost of sequence-level policy gradients varies with length. We find that the advantage magnitude serves as a high-fidelity proxy for gradient norms, and use this to introduce Cost-Aware GRPO. Empirical results on 1.5B, 4B, and 8B LLMs demonstrate that this algorithm significantly reduces the tokens used in policy optimization while matching or exceeding baseline accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。