arXiv:2608.07719cs.LG2026-08

提出可重复使用的离线强化学习数据选择方法,提升数据利用效率。

CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning

  • 基于评论家引导的迭代残差选择,动态筛选高价值数据
  • 10%数据预算下保持96.6%性能,优于多数基线方法
  • 适合追求高效训练与跨任务复用的研究者

离线强化学习从固定数据池中反复训练策略,冗余数据在不同种子和超参数下代价高昂,而简单采样可能剔除长时程信用分配所需的稀有转移。本文提出CODS,一种交替拟合算法匹配评论家并获取高残差转移的评论家引导选择器,最终冻结一个可重复使用的数据子集。不同于优先经验回放,CODS生成静态数据集;不同于单次残差选择,其评分随评论家更新而刷新。在10%预算下,CODS在20个有效D4RL任务-算法组合中保留了96.6%的原始池性能,超越ReDOR和OPER(19/20),且在所有20个子集基线上均表现更优;六项优势在预设分层推断与霍尔姆校正下仍显著。固定总选择更新次数时,五轮采集比单轮提升11.23点并在四组代表性任务中趋于饱和。等价轮次与等价时长评估表明,复用性而非单次加速带来计算优势。机制分析与干扰实验揭示其对稀疏奖励的增益作用及对异常值的敏感性。最后,全轨迹扩展版本在ALFWorld上保留95.4%成功率,在GSM8K上保留96.5%精确匹配率。因此,CODS是一种可重复使用的选择流程,而非形式化的核集合保证。

原文摘要 · Abstract (English)

Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment. We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions before freezing a reusable subset. Unlike prioritized replay, CODS produces a static artifact; unlike one-shot residual selection, it refreshes scores as the critic changes. At a 10\% budget, CODS retains 96.6\% of eligible-pool performance across 20 valid D4RL task--algorithm cells. It exceeds ReDOR and OPER on 19/20 cells and every other subset baseline on 20/20; all six subset advantages remain significant under predeclared hierarchical inference with Holm correction. Holding total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate thereafter. Equal-pass and equal-hour evaluations clarify that reuse, rather than a single-run speedup, creates the compute advantage. Mechanism and corruption interventions expose both useful sparse-reward enrichment and sensitivity to outliers. Finally, a whole-trace extension retains 95.4\% of pooled ALFWorld success and 96.5\% of pooled GSM8K exact match. CODS is therefore a reusable selection procedure, not a formal coreset guarantee.

强化学习数据选择离线学习可复用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。