通过因果与局部性结合，实现大规模网络系统高效可泛化的强化学习。
Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems
- 基于因果掩码提取最小影响邻域，生成紧致状态表示。
- 在多源域上共享策略，仅需少量轨迹即可适应新环境。
- 理论保证收敛性与泛化能力，显著优于传统方法。
大规模网络系统（如交通、电力和无线网络）对强化学习智能体提出规模与环境变化的双重挑战。本文提出GSAC（可泛化且可扩展的演员-评论家框架），将因果表示学习与元演员-评论家学习相结合，兼顾可扩展性与领域泛化。每个智能体首先学习稀疏的局部因果掩码，精确识别影响自身动态的最小邻域变量，从而生成指数级紧凑的状态与领域因子近似表示（ACRs）。这些表示限制了截断价值函数到κ跳邻域的误差，使图结构上的高效学习成为可能。元演员-评论家在多个源域上训练共享策略，并基于紧凑的领域因子进行条件控制；测试时仅需少量轨迹即可估计新领域因子并部署适配策略。本文建立了因果恢复、演员-评论家收敛及适应差距的有限样本保证，结果表明GSAC适应迅速，性能显著优于从头学习和传统自适应基线。
原文摘要 · Abstract (English)
Large-scale networked systems, such as traffic, power, and wireless grids, challenge reinforcement-learning agents with both scale and environment shifts. To address these challenges, we propose GSAC (Generalizable and Scalable Actor-Critic), a framework that couples causal representation learning with meta actor-critic learning to achieve both scalability and domain generalization. Each agent first learns a sparse local causal mask that provably identifies the minimal neighborhood variables influencing its dynamics, yielding exponentially tight approximately compact representations (ACRs) of state and domain factors. These ACRs bound the error of truncating value functions to $κ$-hop neighborhoods, enabling efficient learning on graphs. A meta actor-critic then trains a shared policy across multiple source domains while conditioning on the compact domain factors; at test time, a few trajectories suffice to estimate the new domain factor and deploy the adapted policy. We establish finite-sample guarantees on causal recovery, actor-critic convergence, and adaptation gap, and show that GSAC adapts rapidly and significantly outperforms learning-from-scratch and conventional adaptation baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。