解决延迟反馈下的强化学习偏差问题，提升生产级人类反馈强化学习效果。
Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF
- 引入回溯优势校正机制，将延迟反馈作为残差注入下一轮优化。
- 在双延迟通道场景中，政策偏差降低47.9倍，耗时更低。
- 兼容PPO与GRPO，仅需两行代码修改，适合实际部署的强化学习系统。
生产环境中的基于人类反馈的强化学习（RLHF）常无法获得同步奖励信号。代码执行验证器、慢速评分集成和排队的人类评审可能在生成动作序列后数个梯度步骤才返回结果，破坏了标准PPO所依赖的同步奖励假设。本文提出回溯优势校正（RAC）：将待处理的延迟完成项进行队列管理，通过非负核函数老化，并以截断残差形式重新注入下一优化步的优势计算。证明在无偏截断重要性比条件下，当有效延迟核完全重注入其质量时，累积校正为无偏；否则偏差与未重注入比例呈线性关系；在无延迟身份核情况下退化为V-trace。在一个表格马尔可夫决策过程（MDP）的原型验证中，RAC在双延迟通道配置下将闭式策略偏差降低47.9倍，优于等待延迟结果的方法且墙钟时间更短。RAC可通过两行奖励管理器补丁集成至PPO与GRPO。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) in production does not always have a synchronous reward signal. Code-execution verifiers, slow judge ensembles, and queued human review can return several gradient steps after the rollout that produced them, breaking the synchronous-reward assumption underlying standard PPO. We address this gap with Retroactive Advantage Correction (RAC): each pending slow completion is queued, aged through a non-negative kernel, and reinjected as a clipped residual into the next optimiser step's advantage. We prove that under an unbiased clipped importance ratio, the cumulative RAC correction is exactly unbiased when the effective delay kernel reinjects all of its mass, and carries a bias linear in the unreinjected fraction otherwise; at the no-delay identity kernel it reduces to V-trace. On a tabular Markov decision process (MDP) proof-of-concept, RAC reduces the closed-form policy bias by up to 47.9x at the two-slow-channel configuration, beating wait-for-slow at lower wall-clock cost. RAC integrates with PPO and GRPO through a two-line reward-manager patch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。