让智能体在环境变化时能主动放弃不确定操作,避免错误决策。
Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift

- 设计可选模仿策略,仅在有把握时行动,否则停止
- 新算法在任意环境变化下仍保持低损失,样本效率与时间无关
- 适用于真实部署中专家行为不一致的复杂场景
行为克隆在训练与测试环境动态一致时表现良好,但在实际部署中,测试环境的转移规律常与训练阶段不同。传统离线模仿学习要求智能体在每个状态都必须输出动作,即使示范数据无参考价值,可能导致性能严重下降。为此,本文提出选择性模仿学习框架:在训练环境中几乎不停止,在测试环境中当无法可靠决策时可主动终止。算法SeqRejectron通过一组独立于时序长度和策略类别的验证器策略构建停止规则。对确定性策略,假设代价稀疏,获得不依赖时序的样本复杂度$ ilde{O}("log|Π|/ε^2)$;对随机策略,使用累积赫林格距离作为停止条件,同样实现时序无关的保证。该框架还支持专家模型错配及训练测试中专家行为不一致的情况,性能随错配程度平滑退化。
原文摘要 · Abstract (English)
Behavior cloning provides strong imitation learning guarantees when training and test environments share the same dynamics. However, in many deployment settings the test environment's transitions differ from training, and classical offline IL offers no recourse: the learner must commit to an action at every state, even when its demonstrations are uninformative and could lead to arbitrary degradation of performance. This motivates the study of selective imitation, where the learner may choose to stop when it cannot act reliably. We introduce a model for selective imitation under arbitrary dynamics shift: given labeled expert demonstrations from a training environment and unlabeled state trajectories from the same expert in a test environment, the learner outputs a selective policy that is complete (rarely stops in training) and sound (incurs low regret before stopping in test). Our algorithm, SeqRejectron, constructs a stopping rule using a small set of validator policies whose size is independent of the horizon or policy class. For deterministic policies, this yields horizon-free $\tilde{O}(\log|Π|/ε^2)$ sample complexity, assuming sparse costs. For stochastic policies, we obtain analogous horizon-free guarantees using a cumulative Hellinger stopping time. We extend the framework to misspecified experts and different expert policies across train and test and obtain results that gracefully degrade with the amount of misspecification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。