arXiv:2608.01804cs.LGcs.AI2026-08

针对生成GPU核代码的强化学习难题,提出动态剪枝框架提升效率与稳定性。

LEAP: Lean Environment-Feedback via Adaptive Pruning for Code RL in GPU Kernel Generation

  • 通过自适应剪枝剔除简单或失败任务,聚焦资源于高价值复杂任务。
  • 在CUDA核生成中实现更快收敛,首轮生成成功率更高,多轮调试更鲁棒。
  • 无需调参,基于排名奖励自动优化学习梯度,适合低级系统编程场景。

通过强化学习(RL)后训练大语言模型(LLMs)显著提升了代码生成能力。现有先进框架采用无评价网络范式(如GRPO),结合规则验证沙箱以规避评价网络的高内存开销。然而,在低级系统编程(如CUDA核生成)中面临严峻挑战:二进制通过/失败奖励导致信号稀疏,多轮环境反馈循环受编译延迟高和奖励稀释影响严重。本文提出LEAP(Lean Environment-Feedback via Adaptive Pruning),一种面向低级硬件对齐的可扩展、高效多轮强化学习框架。LEAP引入难度条件剪枝(DCP),一种动态门控机制,自适应地从多轮扩展中剔除简单或过度失败的任务,仅将资源密集型编译与硬件探索用于高价值复杂任务。为实现无手动超参数设计的完整部署,提出基于排名的奖励形式,通过GRPO回放组内成对锦标赛结果推导尺度无关的相对优势,天然惩罚简单提示的词元低效性,同时最大化复杂分布的学习梯度。实证评估表明,LEAP在首轮生成表现更优,具备更强多轮调试鲁棒性,且比未剪枝基线更快收敛,建立了一种实用的低级代码强化学习范式。

原文摘要 · Abstract (English)

Post-training large language models (LLMs) via reinforcement learning (RL) has significantly advanced code generation capabilities. To bypass the heavy memory footprint of critic networks, current state-of-the-art frameworks leverage critic-free paradigms like Group Relative Policy Optimization (GRPO) tied to rule-based verification sandboxes. However, applying these frameworks to low-level systems programming, such as CUDA kernel generation-presents severe challenges: binary pass/fail rewards introduce severe signal sparsity, while multi-turn environmental feedback loops suffer from prohibitive compilation latencies and reward dilution across trajectories. In this work, we introduce LEAP (Lean Environment-Feedback via Adaptive Pruning), a scalable and computationally efficient multi-turn RL framework optimized for low-level hardware accelerator alignment. LEAP features Difficulty-Conditioned Pruning (DCP), a dynamic gating mechanism that adaptively cuts off simple and overly catastrophic tasks from multi-turn expansion, focusing resource-heavy compilation and hardware exploration exclusively on high-value, complex tasks. To fully operationalize these paths without manual hyperparameter engineering, we propose a Rank-Based Reward formulation. By deriving scale-free relative advantages from pairwise tournament outcomes within the GRPO rollout group, our method inherently penalizes token inefficiency on simple prompts while maximizing learning gradients on challenging distributions. Empirical evaluations show that LEAP achieves superior first-turn proficiency and robust multi-turn debugging resilience while converging faster than unpruned multi-turn baselines, establishing a practical paradigm for low-level code RL.

代码生成强化学习CUDA核自适应剪枝

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。