用非参数贝叶斯方法从多专家数据中识别不同奖励类型,自动确定专家数量。
Nonparametric Bayesian Inverse Reinforcement Learning with Data-Parallel Gibbs Sampling

- 基于狄利克雷过程建模奖励函数,自动推断潜在专家类型数
- 在10x10 ObjectWorld上准确识别出K=2和K=3真实奖励类型
- 支持并行计算,适合高并发场景下的多专家行为分析
逆强化学习从专家示范中恢复奖励函数,但传统方法假设所有示范来自单一专家。当示范来自具有不同偏好的多位专家时,参数化方法会得到一个平均奖励,无法拟合任一专家。本文采用狄利克雷过程先验的非参数贝叶斯逆强化学习,使潜在奖励类型数与奖励本身共同推断。推理使用融合中国餐馆过程聚类更新与马尔可夫链蒙特卡洛采样的折叠吉布斯采样器,内层规划采用软值迭代。在10x10 ObjectWorld网格环境中,串行采样对K=2的情况达到调整兰德指数1.000,显著优于最大熵逆强化学习基线(ARI=0.000)。扩展至K=3时,采样器在所有运行中正确识别出聚类数;聚类分配的ARI为0.48-0.58,反映不同专家行为存在持续重叠,表明在ObjectWorld上可靠评估K=3需控制物体放置而非随机初始化。进一步通过Ray在高性能计算硬件上实现跨CPU核心并行,8个工作进程下峰值加速比达4.79倍,并揭示状态聚合中共识合并启发式带来的吞吐量与精度权衡。代码与容器环境已开源:https://github.com/dasashreeya/np_bayes_irl。
原文摘要 · Abstract (English)
Inverse Reinforcement Learning recovers reward functions from expert demonstrations, but standard formulations assume that all demonstrations come from a single expert. When demonstrations are pooled from multiple experts with distinct preferences, parametric methods recover an averaged reward that fits no individual expert well. We implement Nonparametric Bayesian Inverse Reinforcement Learning with a Dirichlet Process prior over reward functions, allowing the number of latent reward types to be inferred jointly with the rewards themselves. Inference uses a collapsed Gibbs sampler combining a Chinese Restaurant Process update for cluster assignments with a Metropolis-Hastings update for reward weights, and soft value iteration as the inner planning routine. We evaluate on a 10x10 ObjectWorld grid with two and three ground-truth reward types. The serial sampler recovers K=2 with Adjusted Rand Index of 1.000, substantially outperforming a Maximum Entropy IRL baseline (ARI=0.000). Extension to K=3 shows that the sampler correctly identifies the number of clusters in all runs; assignment ARI of 0.48-0.58 reflects behavioral overlap between expert types that persists across grid instantiations, revealing that reliable K=3 evaluation on ObjectWorld requires controlled object placement rather than random seeding. We further parallelize the sampler across CPU cores using Ray on HPC hardware, achieving a peak speedup of 4.79x at 8 workers, and characterize a throughput-versus-accuracy tradeoff arising from the consensus merge heuristic used during state aggregation. Code and a containerized environment are available at https://github.com/dasashreeya/np_bayes_irl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。