用大模型指导直接调整Q值，提升强化学习样本效率。
From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledge
- 以大模型提供启发式信息，直接初始化和修正Q值
- 在20个环境上比最优基线提升16.87%样本效率
- 比基于奖励塑形的方法提升253.80%，更公平无偏
Q-shaping 是 Q 值初始化的扩展，作为奖励塑形的替代方法，通过直接塑造 Q 值来融入领域知识，从而加速智能体训练并提升样本效率。该方法在多种任务中具有通用性和鲁棒性，可即时评估效果并保证最优性。我们在20个不同环境中使用大语言模型（LLM）作为启发式提供者进行评估。结果表明，Q-shaping 显著提升了样本效率，在每个环境中的表现均优于最佳基线16.87%，相比基于 LLM 的奖励塑形方法更是提升了253.80%。这些发现确立了 Q-shaping 在强化学习中作为更优、更无偏的奖励塑形替代方案的地位。
原文摘要 · Abstract (English)
Q-shaping is an extension of Q-value initialization and serves as an alternative to reward shaping for incorporating domain knowledge to accelerate agent training, thereby improving sample efficiency by directly shaping Q-values. This approach is both general and robust across diverse tasks, allowing for immediate impact assessment while guaranteeing optimality. We evaluated Q-shaping across 20 different environments using a large language model (LLM) as the heuristic provider. The results demonstrate that Q-shaping significantly enhances sample efficiency, achieving a \textbf{16.87\%} improvement over the best baseline in each environment and a \textbf{253.80\%} improvement compared to LLM-based reward shaping methods. These findings establish Q-shaping as a superior and unbiased alternative to conventional reward shaping in reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。