从行为推断偏好,用经济理论和随机算法重构智能体的决策逻辑。
Inverse Reinforcement Learning using Revealed Preferences and Passive Stochastic Optimization
- 基于微观经济学的揭示偏好理论,从动作反推效用函数
- 在噪声干扰下仍能检测理性决策行为,准确识别认知雷达等智能体
- 提出可实时追踪变化效用的自适应算法,适合动态环境研究者
本专著分三章探讨逆强化学习(IRL)。第一章以微观经济学中的揭示偏好理论(Afriat定理及扩展)为视角,基于观测到的智能体行为识别约束效用最大化者,并重建其效用函数的集合估计。通过该方法成功识别出认知雷达的存在并重构其效用函数,同时提出在动作受噪声干扰时的统计效用最大化检测器。第二章研究贝叶斯逆强化学习,探讨分析者如何判断观察到的智能体是否为同时优化效用与观测似然的理性不注意贝叶斯最大化者,并讨论逆停止时间问题,重构贝叶斯智能体在随机时域下的继续与停止成本。进一步应用此方法识别贝叶斯最优序贯探测器。本章还简要综述离散选择模型、逆贝叶斯滤波及用于自适应IRL的逆随机梯度算法。第三章引入被动朗之万动力学的自适应IRL方法,旨在面对噪声和误设梯度时追踪时变效用函数。本质上,第三章提出的自适应算法可视为逆随机梯度算法,因其在随机梯度算法运行时实时学习效用函数。
原文摘要 · Abstract (English)
This monograph, spanning three chapters, explores Inverse Reinforcement Learning (IRL). The first two chapters view inverse reinforcement learning (IRL) through the lens of revealed preferences from microeconomics while the third chapter studies adaptive IRL via Langevin dynamics stochastic gradient algorithms. Chapter uses classical revealed preference theory (Afriat's theorem and extensions) to identify constrained utility maximizers based on observed agent actions. This allows for the reconstruction of set-valued estimates of an agent's utility. We illustrate this procedure by identifying the presence of a cognitive radar and reconstructing its utility function. The chapter also addresses the construction of a statistical detector for utility maximization behavior when agent actions are corrupted by noise. Chapter 2 studies Bayesian IRL. It investigates how an analyst can determine if an observed agent is a rationally inattentive Bayesian utility maximizer (i.e., simultaneously optimizing its utility and observation likelihood). The chapter discusses inverse stopping-time problems, focusing on reconstructing the continuation and stopping costs of a Bayesian agent operating over a random horizon. We then apply this IRL methodology to identify the presence of a Bayes-optimal sequential detector. Additionally, Chapter 2 provides a concise overview of discrete choice models, inverse Bayesian filtering, and inverse stochastic gradient algorithms for adaptive IRL. Finally, Chapter 3 introduces an adaptive IRL approach utilizing passive Langevin dynamics. This method aims to track time-varying utility functions given noisy and misspecified gradients. In essence, the adaptive IRL algorithms presented in Chapter 3 can be conceptualized as inverse stochastic gradient algorithms, as they learn the utility function in real-time while a stochastic gradient algorithm is in operation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。