arXiv:2509.09219cs.LGcs.AI2025-09

用图神经网络实现可泛化的强化学习决策,适配不同规模的复杂状态。

Vejde: A Framework for Inductive Deep Reinforcement Learning Based on Factor Graph Color Refinement

  • 将状态建模为实体事实的二分图,通过消息传递提取潜在表示
  • 在8个RDDL问题上平均表现接近专用MLP模型,且无需重新训练
  • 适合需要跨结构泛化的强化学习任务,如多智能体或动态环境

我们提出并评估了Vejde:一种结合数据抽象、图神经网络与强化学习的框架,用于生成具有丰富结构状态(如对象类别和关系)的决策问题的归纳策略函数。马尔可夫决策过程的状态被表示为实体事实的数据库,Vejde将每个状态转换为二分图,并通过神经消息传递映射到潜在状态。状态与动作的因子化表示使Vejde代理能够处理不同规模和结构的问题。我们在8个由RDDL定义的问题领域中测试了Vejde代理,每个领域包含10个问题实例,策略分别通过监督学习和强化学习训练。为测试策略泛化能力,我们将问题实例分为训练集和仅用于测试的集合。对未见实例的测试结果与针对每个实例训练的MLP代理以及在线规划算法Prost进行比较。结果显示,Vejde策略在平均情况下对未见实例保持良好性能,得分无显著下降;且在未见测试实例上的平均得分接近于特定实例训练的MLP代理。

原文摘要 · Abstract (English)

We present and evaluate Vejde; a framework which combines data abstraction, graph neural networks and reinforcement learning to produce inductive policy functions for decision problems with richly structured states, such as object classes and relations. MDP states are represented as data bases of facts about entities, and Vejde converts each state to a bipartite graph, which is mapped to latent states through neural message passing. The factored representation of both states and actions allows Vejde agents to handle problems of varying size and structure. We tested Vejde agents on eight problem domains defined in RDDL, with ten problem instances each, where policies were trained using both supervised and reinforcement learning. To test policy generalization, we separate problem instances in two sets, one for training and the other solely for testing. Test results on unseen instances for the Vejde agents were compared to MLP agents trained on each problem instance, as well as the online planning algorithm Prost. Our results show that Vejde policies in average generalize to the test instances without a significant loss in score. Additionally, the inductive agents received scores on unseen test instances that on average were close to the instance-specific MLP agents.

强化学习图神经网络归纳推理RDDL

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。