arXiv:2608.01205cs.LGcs.AI2026-08

用简单方法复现最强性能,仅调两个参数就超越多数复杂模型。

ReBRAC-v2: The Return of the King

论文配图:ReBRAC-v2: The Return of the King
图 1 · 摘自论文原文
  • 用精确似然流做策略,结合多种正则化与分类残差评论器。
  • 在10个类别上平均得分74.8,领先第二名22.5分,8类第一。
  • 固定结构只调参数,适配多个任务,适合追求简洁高效的读者。

近期离线强化学习方法越来越多依赖于表达性强的生成策略和专用的价值引导机制。我们提出是否可通过系统性地现代化传统行为正则化演员-评论家框架,在保持算法简洁的同时实现类似进展。本文提出ReBRAC-v2:直接使用精确似然归一化流作为强化学习演员,融合似然、MSE和MAE行为正则化,引入基于分类的残差评论器、分阶段优化和多样本测试时动作选择。不针对每项任务单独调参,而是通过约600次贝叶斯建议在六个挑战性OGBench任务上确定单一共享配置,冻结所有结构与优化设置,仅在16点网格上调整两个行为正则化系数。在十个常见状态基OGBench类别中,ReBRAC-v2平均得分为74.8,优于次优结果52.3,八个类别排名第一。同一配置在D4RL AntMaze(90.2)和Adroit(33.6)上也取得最强平均表现。固定配方消融实验表明,最大敏感性来自混合克隆目标、分阶段训练、足够容量的流模型及多样本推理;而其他小选择则依赖于其他超参数取值。结果表明,有纪律的可迁移工程可在不放弃极简离线强化学习基础的前提下实现最先进综合性能。

原文摘要 · Abstract (English)

Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value-guidance mechanisms. We ask whether comparable progress can instead come from systematically modernizing a conventional behavior-regularized actor-critic while preserving its algorithmic simplicity. We introduce ReBRAC-v2, which directly trains an exact-likelihood normalizing flow as the RL actor, combines likelihood, MSE, and MAE behavior regularization, and integrates a classification-based residual critic, staged optimization, and multi-sample test-time action selection. Rather than tuning this recipe separately for every task, we develop a single shared configuration via roughly 600 Bayesian proposals on six challenging OGBench tasks, freeze all structural and optimization choices, and adapt only two behavior-regularization coefficients over a 16-point grid. Across ten common state-based OGBench categories, ReBRAC-v2 averages 74.8 compared to 52.3 for the next-best aggregate result and ranks first in eight categories. The same recipe, without structural changes, obtains the strongest averages in our comparisons on D4RL AntMaze (90.2) and Adroit (33.6). Fixed-recipe ablations show the largest sensitivity to the selected mixed cloning objective, staged training, sufficient flow capacity, and multi-sample inference, while showing that several smaller choices depend on the values of other hyperparameters. These results show that disciplined, transferable engineering can achieve state-of-the-art aggregate performance without abandoning a minimalist offline RL foundation.

离线强化学习行为正则化归一化流可迁移设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。