arXiv:2605.06111cs.SEcs.AI2026-05

用任务价值动态调度训练,让大模型同时学好多种编程任务。

Schedule-and-Calibrate: Utility-Guided Multi-Task Reinforcement Learning for Code LLMs

论文配图:Schedule-and-Calibrate: Utility-Guided Multi-Task Reinforcement Learning for Code LLMs
图 1 · 摘自论文原文
  • 根据任务学习潜力和协同效应动态分配训练资源。
  • 在4个编程任务上优于单任务专家模型9.0%~9.5%。
  • 适合需要统一模型处理多类编程任务的场景。

基于可验证奖励的强化学习在代码大模型后训练中表现优异,但为每项任务单独部署专用模型会导致成本随任务数增加而上升,促使采用统一的多任务强化学习(MTRL)方法。然而,现有MTRL方法对所有编程任务一视同仁,依赖固定数据教学计划和统一优化策略,限制了多任务训练效果。为此,我们提出ASTOR:一种基于任务价值驱动协调的多任务代码强化学习框架。其核心是任务价值信号,用于衡量各任务的学习潜力与跨任务协同效应。ASTOR包含两个耦合模块:1)分层任务价值路由数据调度模块,分层分配训练预算并优先选择高价值提示;2)自适应任务价值校准策略优化模块,动态调整每项任务的KL正则化强度,使其更新约束匹配当前训练状态。在两种主流大模型上针对四个代表性编程任务的实验表明,ASTOR始终能提升单一模型在所有任务上的表现,优于最优单任务专家模型9.0%~9.5%,超越最强的MTRL基线7.5%~12.8%。

原文摘要 · Abstract (English)

Reinforcement learning (RL) with verifiable rewards has proven effective at post-training LLMs for coding, yet deploying separate task-specific specialists incurs costs that scale with the number of tasks, motivating a unified multi-task RL (MTRL) approach. However, existing MTRL methods treat all coding tasks uniformly, relying on fixed data curricula under a shared optimization strategy, ultimately limiting the effectiveness of multi-task training. To address these limitations, we propose ASTOR, a multi-tASk code reinforcement learning framework via uTility-driven coORdination. Centered on task utility, a signal capturing each task learning potential and cross-task synergy, ASTOR comprises two coupled modules: 1) Hierarchical Utility-Routed Data Scheduling module hierarchically allocates training budget and prioritizes informative prompts, steering training toward the most valuable data and 2) Adaptive Utility-Calibrated Policy Optimization module dynamically scales per-task KL regularization, matching update constraints to each tasks current training state. Experiments on two widely-used LLMs across four representative coding tasks demonstrate that ASTOR consistently improves a single model across all tasks, outperforming the best task-specific specialist by 9.0%-9.5% and surpassing the strongest MTRL baseline by 7.5%-12.8%.

强化学习代码生成多任务学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。