arXiv:2608.28421cs.AI2026-08

让大模型学会可验证的编程推理,结果透明可检查。

Program Learning with Verifiable Rewards: Symbolic Backpropagation for Post-Training LLMs

  • 用符号反向传播学习可验证程序,每步都有明确反馈
  • 30B模型在两个基准上比强化学习高27.8分
  • 只需100个示例即可适配新任务,无需额外微调数据

将语言模型用于推理意味着更新其权重。监督微调和强化学习都将能力内置于模型权重中,无法观测、无法逐步验证,也无法迁移至其他模型。我们主张:对于中间步骤可验证的任务,推理应置于基础模型之外,作为由确定性与神经原语构成的显式程序。本文提出PLVR(Program Learning with Verifiable Rewards):一种从输入输出样例直接学习此类程序的后训练方法。其核心机制为符号反向传播:每层程序携带类型本体,输出损失基于真实标签计算,通过原语签名的类型推断反向传播所需输入本体——类似链式法则,信用分配是推导而非估算。与仅验证最终结果的RLVR不同,PLVR的奖励是稠密分布于程序结构中的每步合同判定。在LiveCodeBench v6和Tau2Bench上,30B基模型使用PLVR在相同预算下平均领先强化学习27.8分,前沿模型(大一个数量级)领先13.6分。单一原语库服务于两个基准,新任务边际成本仅为100个程序搜索示例,无需新增微调数据。在相同预算下,若将损失引导搜索替换为均匀采样同一类型允许空间,程序中位性能从65.6降至17.5,表明优势源于反向传播而非类型系统。作者发布符号反向传播库和合规性检查器,支持其他原语库应用。

原文摘要 · Abstract (English)

Post training a language model to reason means updating its weights. Supervised finetuning and reinforcement learning both place the acquired capability inside the model where it cannot be inspected cannot be checked step by step and cannot be moved to another model. We argue that for tasks whose intermediate steps admit verification, reasoning is better placed outside the base models weights as an explicit program composed from deterministic and neural primitives. We introduce PLVR (Program Learning with Verifiable Rewards): a post training method that learns such programs directly from input-output examples. Its mechanism is symbolic backpropagation: each program layer carries a typed ontology a loss is computed at the output against ground truth and required input ontologies are propagated backward by type inference over primitive signatures: an analogue of the chain rule in which credit assignment is a derivation rather than an estimate. Where RLVR verifies a terminal outcome, PLVRs reward is a per step contract verdict dense over program structure. On LiveCodeBench v6 and Tau2Bench, 30B base models with PLVR outperform RL at matched budget by 27.8 points on average and frontier models an order of magnitude larger by 13.6 points. A single primitive library serves two benchmarks, so the marginal cost of a new task is 100 examples of program search and no new finetuning data. Replacing the loss guided search with uniform sampling over the same type admissible space at equal budget collapses the median program from 65.6 to 17.5, identifying the backward pass rather than the type system as the source of the advantage. We release the symbolic backpropagation library and a conformance checker so the method can be applied to primitive libraries other than our own.

程序学习符号推理可验证性后训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。