通过分离观测与控制信息，提升强化学习在未知环境中的泛化能力
Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalization
- 将上下文学习拆解为观测和控制两个层次，分别优化
- 在参数变化的连续控制任务中，用更少样本达到更好性能
- 适合需要强泛化能力的机器人控制场景
捕捉潜在变化（即‘上下文’）是将强化学习代理部署到训练范围之外的关键。我们将基于上下文的强化学习重新建模为一个双推理-控制问题，并形式化定义了两个性质及其层级关系：观测充分性（保留所有预测信息）和控制充分性（保留决策相关的信息）。利用这一二分法，我们推导出一种类似上下文证据下界（ELBO）的目标函数，可清晰分离表征学习与策略学习，并通过瓶颈式上下文策略优化（BCPO）算法实现优化——该算法在任意离策略策略学习器前加入变分信息瓶颈编码器。在具有参数漂移的标准连续控制基准测试中，BCPO在样本量更少的情况下表现优于或媲美其他基线方法，且在训练范围外仍保持良好性能。该框架统一了基于上下文强化学习的理论、诊断与实践。
原文摘要 · Abstract (English)
Capturing latent variations ("contexts") is key to deploying reinforcement-learning (RL) agents beyond their training regime. We recast context-based RL as a dual inference-control problem and formally characterize two properties and their hierarchy: observation sufficiency (preserving all predictive information) and control sufficiency (retaining decision-making relevant information). Exploiting this dichotomy, we derive a contextual evidence lower bound(ELBO)-style objective that cleanly separates representation learning from policy learning and optimizes it with Bottlenecked Contextual Policy Optimization (BCPO), an algorithm that places a variational information-bottleneck encoder in front of any off-policy policy learner. On standard continuous-control benchmarks with shifting physical parameters, BCPO matches or surpasses other baselines while using fewer samples and retaining performance far outside the training regime. The framework unifies theory, diagnostics, and practice for context-based RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。