提出一种分布映射方法，让强化学习更公平地对待不同人群。
A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning
- 用分位数分布映射预处理数据，估计反事实状态和奖励。
- 理论证明每步不公平性与长期损失均有界，条件温和。
- 在真实医疗数据上验证有效，适合高风险决策场景使用。
强化学习旨在优化序列决策以提升整体长期效益。但在医疗等高风险场景中，其决策可能系统性地限制某些子群体获取有价值服务，违背利益相关方价值。反事实公平性（CF）基于因果推理提供解决思路。本文提出一种数据预处理算法，与策略学习结合可实现强化学习中的反事实公平性。该算法采用新颖的分位数分布映射方法，在预处理阶段顺序估计反事实状态与奖励，包含常见的加性假设作为特例。理论上证明，在较弱正则性条件下，每步反事实不公平性与无限时域次优差距均可被界定。我们在数值实验及真实干预型数字健康数据集上进行了实证测试。
原文摘要 · Abstract (English)
Reinforcement learning (RL) seeks to optimize sequential decisions to maximize population-level benefits over time. However, when deployed in high-stakes settings such as healthcare, RL decisions might systematically restrict some subpopulation's access to valuable services in a manner contrary to the values and goals of stakeholders. Counterfactual fairness (CF) offers a promising framework to address this problem based on causal reasoning. This paper develops a data preprocessing algorithm that, when used in tandem with policy learning, enables CF in RL. Our algorithm relies on a novel quantile distribution mapping method for sequentially estimating the counterfactual states and rewards in the data preprocessing step, subsuming common additivity assumptions used for counterfactual prediction as a special case. We theoretically prove that the per-step level of counterfactual unfairness and infinite-horizon suboptimality gap can be bounded under mild regularity conditions. We also empirically test our algorithm in numerical experiments as well as in application to a real-world interventional digital health dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。