用社会选择理论融合多人不同策略，生成集体最优决策。
Policy Aggregation
- 将个体偏好转化为状态-动作占用多面体的体积比较
- 实现多种投票机制在策略聚合中的实际应用
- 适合多用户协同决策场景的公平性优化
我们研究多个具有不同奖励函数和最优策略的个体在马尔可夫决策过程中的人工智能价值对齐问题。该问题被形式化为策略聚合，目标是识别出理想的集体策略。我们认为基于社会选择理论的方法尤为合适。关键洞察在于，可将序数偏好重新解释为状态-动作占用多面体子集的体积。基于此，我们证明了多种方法——包括同意投票、波达计数、比例否决核心和分位数公平性——可实际应用于策略聚合。
原文摘要 · Abstract (English)
We consider the challenge of AI value alignment with multiple individuals that have different reward functions and optimal policies in an underlying Markov decision process. We formalize this problem as one of policy aggregation, where the goal is to identify a desirable collective policy. We argue that an approach informed by social choice theory is especially suitable. Our key insight is that social choice methods can be reinterpreted by identifying ordinal preferences with volumes of subsets of the state-action occupancy polytope. Building on this insight, we demonstrate that a variety of methods--including approval voting, Borda count, the proportional veto core, and quantile fairness--can be practically applied to policy aggregation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。