不依赖奖励信号，仅通过观察行为就能恢复最优策略。
Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitation
- 用两阶段后缀模仿框架，舍弃初期探索数据，仅用后期稳定行为训练。
- 在无奖励条件下仍达到 $\tilde O(1/\sqrt{N})$ 的收敛速度。
- 适合研究行为逆向建模或隐私敏感场景的算法设计者。
我们研究逆上下文老虎机（Inverse Contextual Bandit, ICB）问题：学习者试图优化策略，而观察者无法获取学习者的奖励信息，仅能观测其动作，目标是恢复底层问题参数。学习过程中，学习者的行为自然从探索转向利用，导致动作数据非平稳，给观察者带来重大挑战。为此，我们提出简单有效的两阶段后缀模仿框架：舍弃初始预热阶段数据，仅使用后续模仿阶段的数据进行经验风险最小化。我们推导出预测决策损失的界，明确刻画了预热长度选择带来的偏差-方差权衡。尽管存在严重信息缺失，我们证明奖励无关的观察者仍可实现 $\tilde O(1/\sqrt{N})$ 的收敛速率，与完全知晓奖励的学习者渐近效率相当。结果表明，被动观察者仅凭动作序列即可有效还原最优策略，性能接近学习者自身。
原文摘要 · Abstract (English)
We study the Inverse Contextual Bandit (ICB) problem, in which a learner seeks to optimize a policy while an observer, who cannot access the learner's rewards and only observes actions, aims to recover the underlying problem parameters. During the learning process, the learner's behavior naturally transitions from exploration to exploitation, resulting in non-stationary action data that poses significant challenges for the observer. To address this issue, we propose a simple and effective framework called Two-Phase Suffix Imitation. The framework discards data from an initial burn-in phase and performs empirical risk minimization using only data from a subsequent imitation phase. We derive a predictive decision loss bound that explicitly characterizes the bias-variance trade-off induced by the choice of burn-in length. Despite the severe information deficit, we show that a reward-free observer can achieve a convergence rate of $\tilde O(1/\sqrt{N})$, matching the asymptotic efficiency of a fully reward-aware learner. This result demonstrates that a passive observer can effectively uncover the optimal policy from actions alone, attaining performance comparable to that of the learner itself.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。