用实验补足模拟器短板,让决策更贴近真实世界。
Mind the Sim-to-Real Gap & Think Like a Scientist

- 区分模拟误差来源,用随机化识别数据偏移部分。
- 模拟最优策略与真实最优间差距由可访问状态和不可达状态决定。
- 提出Fisher-SEP方法,主动设计实验提升预测准确性。
假设规划者拥有一个预训练的序列决策问题模拟器,并可选择在真实环境中进行实验。模拟器查询成本低,但其校准数据引入混淆与漂移;实验无偏但每次消耗一个真实样本。我们研究何时以及如何用实验补充模拟器。首先,扩展的模拟引理将模拟器的价值误差分解为:可通过随机化识别的校准-部署偏移,以及无法通过进一步交互消除的参数残差。其次,模拟最优策略与最优策略之间的价值差距分为局部成分(已访问状态)和可达性成分(未访问状态),后者在纯被动学习下始终不收敛于零。第三,提出Fisher-SEP,一种基于模拟辅助的实验策略,最小化目标策略价值的后验预测方差,包含仅奖励与仅转移的特化版本。两个案例研究说明:在自动售货机供应链中,前期集中实验在长时域下优于后续更新;在艾滋病移动检测场景中,仅设计探索能抵达监控薄弱区域。
原文摘要 · Abstract (English)
Suppose a planner has a pre-trained simulator of a sequential decision problem and the option to run real experiments in the field. The simulator is cheap to query but inherits confounding and drift from its calibration data. Experimentation is unbiased but consumes one real unit per trial. We study when, and how, the planner should supplement the simulator with experiments. We give three results. First, an extended simulation lemma decomposes the simulator's value error into a calibration--deployment shift that randomization can identify and a parametric residual that no further interaction can reduce. Second, the value gap between the simulator-optimal policy and the optimum splits into a local component, on states the deployed policy already visits, and a reachability component, on states it does not. The reachability component stays bounded away from zero at any horizon under purely passive learning. Third, we propose Fisher-SEP, a simulation-aided experimental policy (SEP) that minimizes the posterior predictive variance of a target policy's value, with reward-only and transition-only specializations. Two case studies illustrate the regimes. In a vending-machine supply chain, front-loaded experimentation overtakes posterior updating once the horizon is long enough to amortize the pilot. In an HIV mobile-testing example with a corridor that separates a well-surveilled region from a poorly-surveilled one, only designed exploration reaches the poorly-surveilled region.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。