用符号逻辑提升强化学习采样效率，让智能体更快学会复杂任务。
Sample-efficient Neuro-symbolic Proximal Policy Optimization
- 结合符号规则与PPO，在采样时引入逻辑约束引导动作选择。
- 在三个基准环境上，学习速度提升30%以上，最终奖励更高。
- 适合需要少样本、长周期规划的复杂任务场景。
深度强化学习算法通常需要大量数据，且在稀疏奖励、长规划周期和多子目标的环境中表现不佳。本文提出一种神经符号化的近端策略优化（PPO）扩展方法，将简单任务中学习到的部分逻辑策略规范迁移至更复杂场景以指导学习。提出两种符号引导集成方式：(i) H-PPO-Product，在采样时偏置动作分布；(ii) H-PPO-SymLoss，通过符号正则化项增强PPO损失函数。在OfficeWorld、WaterWorld和DoorKey三个基准测试中，我们的方法均显著优于PPO和奖励机器基线，学习速度更快，收敛回报更高，即使在符号知识不完整的情况下也保持有效。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (DRL) algorithms often require a large amount of data and struggle in sparse-reward domains with long planning horizons and multiple sub-goals. In this paper, we propose a neuro-symbolic extension of Proximal Policy Optimization (PPO) that transfers partial logical policy specifications learned in easier instances to guide learning in more challenging settings. We introduce two integrations of symbolic guidance: (i) H-PPO-Product, which biases the action distribution at sampling time, and (ii) H-PPO-SymLoss, which augments the PPO loss with a symbolic regularization term. We evaluate our methods on three benchmarks (OfficeWorld, WaterWorld, and DoorKey), showing consistently faster learning and higher return at convergence than PPO and a Reward Machine baseline, also under imperfect symbolic knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。