arXiv:2607.00164cs.LG2026-07被引 1

用可验证的奖励训练出精准预测比赛胜率的模型,无需人工标注。

Verifiable Rewards for Calibrated Probabilistic Forecasting

  • 设计无标签的实时胜率奖励,消除结果噪声干扰。
  • 70亿参数模型仅靠自监督训练,校准度媲美投注市场。
  • 保留推理链不被梯度破坏,适合需要可信预测的场景。

基于可验证奖励的强化学习理论上可训练出校准的概率预测模型,因为像布里尔得分这类正确评分规则仅依赖实际结果,且在期望上由真实概率最小化。但实践中会破坏校准性;现有方法主要解决认知不确定性(即模型信心与可验证对错一致的情况)。本文研究随机性预测,其中预测本身是输出,标签是单一随机结果,以美国职业橄榄球联赛(NFL)比赛中场胜率作为测试基准,以投注市场为参考标准。直接奖励实际每回合结果失败,因单个结果噪声大,策略梯度会污染推理过程。为此提出一种可验证、无标签的奖励:基于状态的历史胜率估计,消除标签噪声;并通过直接预测或梯度掩码,使梯度不作用于推理链,避免污染。仅使用该奖励训练,无需人类标签或监督微调,一个70亿参数模型通过直接预测达到与投注市场相当的校准度,且优于零样本前沿模型。该前沿模型与表格估算器达到相同的布里尔得分,表明市场仍存在微小优势,源于它们未共享的实时信息。相比丢弃推理链,梯度掩码保留了推理过程,而普通链式思维训练会破坏它。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards can in principle train calibrated probabilistic forecasters, since a proper scoring rule such as the Brier score is computed from outcomes alone and is minimized in expectation by the true probability. In practice it degrades calibration, and existing remedies address epistemic uncertainty, where a model's confidence accompanies a verifiably correct or incorrect answer. We study aleatoric forecasting, where the forecast itself is the output and the label is one stochastic outcome, taking NFL in-game win probability as a testbed with the betting market as a reference. Rewarding the realized per-play outcome fails, because the single outcome is a noisy target and the policy gradient corrupts the chain of thought. We introduce a verifiable, label-free reward, a state-conditioned empirical win rate estimated from past outcomes, that removes the label noise, and we keep the gradient off the reasoning, by direct prediction or a gradient mask, so it cannot be corrupted. Trained with this reward alone, without human labels or supervised fine-tuning, a 7B model reaches the calibration of the betting market by direct prediction and is better calibrated than a zero-shot frontier model. That frontier model and a tabular estimator reach the same Brier score as this model, identifying the market's small remaining edge as live in-game information beyond their shared inputs. Masking the gradient, rather than dropping the chain of thought, preserves reasoning from which the forecast follows, which ordinary chain-of-thought training corrupts.

概率预测强化学习校准无监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。