让大模型主动提炼经验规则,提升强化学习的数据效率。
From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay

- 用大模型零样本提取行为规则,转化为可微逻辑表示。
- 规则驱动重加权经验回放,在多类任务中提速收敛。
- 适合研究智能体知识构建与高效强化学习的学者。
尽管经验回放对强化学习中的数据效率至关重要,但传统方法将重放缓冲区视为被动记忆系统,仅根据数值预测误差优先选择样本,这与人类通过主动抽象碎片化经验形成行为规则来加速掌握的方式相悖。为弥合这一差距,我们提出神经符号经验回放(NSER),将经验回放从被动样本复用机制转变为活跃的知识构建引擎。具体而言,NSER通过新颖的神经符号接地流程,解决语言推理与数值优化之间的不兼容问题。它以零样本方式利用大语言模型(LLMs)从积累的轨迹中推导候选行为规则,将这些见解接地为可微的一阶逻辑表示,并利用生成的符号结构动态重加权回放分布。通过允许抽象知识直接塑造策略优化,NSER在反应式、基于规则和程序性基准上均实现一致更优的样本效率和收敛速度。
原文摘要 · Abstract (English)
While experience replay is essential for data efficiency in reinforcement learning (RL), standard methods treat the replay buffer as a passive memory system, prioritizing samples based on numerical prediction errors rather than their semantic significance. This approach stands in contrast to human learning, which accelerates mastery by actively abstracting fragmented experiences into behavioral rules. To bridge this gap, we propose Neuro-Symbolic Experience Replay (NSER), a framework that transforms experience replay from a passive sample reuse mechanism into an active engine for knowledge construction. Specifically, NSER addresses the incompatibility between linguistic reasoning and numerical optimization through a novel neuro-symbolic grounding pipeline. It leverages Large Language Models (LLMs) in a zero-shot manner to induce candidate behavioral rules from accumulated trajectories, grounds these insights into differentiable first-order logic representations, and utilizes the resulting symbolic structures to dynamically reweight the replay distribution. By allowing abstract knowledge to directly shape policy optimization, NSER achieves consistent superior sample efficiency and convergence speed across reactive, rule-based, and procedural benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。