用小语言模型慢思考，让强化学习快反应，提升应对陌生环境能力
When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning
- 混合架构：快反应策略+慢思考小模型规划
- 在3个渐难冻结湖环境中超越所有基线方法
- 仅用20亿参数小模型实现安全可行的计划执行
强化学习策略在陌生环境中常因缺乏显式思考而表现下降。本文提出Plan, Align, Commit, Think（PACT）混合架构，将快速反应型强化学习策略与慢速、有思考能力的小语言模型（SLM）规划器结合。PACT异步调用SLM生成并验证候选行动方案；一旦通过仿真验证为安全、可行且完整，便直接执行该计划，无需重新训练或修改原有强化学习策略。在三个难度递增的FrozenLake配置上评估，PACT在仅使用20亿参数的SLM基础上，全面优于所有基线方法，表明在这些场景下，规划性思考与反应式执行协同工作比单一方式更有效。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) policies often degrade in unfamiliar environments because they lack explicit deliberation. We propose Plan, Align, Commit, Think (PACT), a hybrid architecture that combines a fast, reactive RL policy with a slow, deliberative Small Language Model (SLM) planner. PACT invokes the SLM asynchronously to generate and validate candidate action plans. Once a plan is verified through simulation as safe, feasible, and complete, it is executed directly, bypassing the RL policy without retraining or modifying it. Evaluated on three FrozenLake configurations of increasing difficulty, PACT outperforms all baselines while relying on a 2B-parameter SLM backbone, suggesting that deliberative planning and reactive execution are more powerful in concert than either is alone in these settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。