让大模型与强化学习互相进化，解决大规模决策难题
Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
- 大模型充当策略和价值评估双重角色，提升决策质量
- 在超6万动作空间中实现更高效学习，性能超越现有方法
- 适合电力系统等高复杂度工业决策场景
大型语言模型（LLMs）和强化学习（RL）在决策任务中展现出巨大潜力。然而，在大规模工业决策问题中，两者各面临挑战：LLMs缺乏实时长序列决策能力，而RL在庞大动作空间中样本效率低。为此，我们提出大模型与强化学习协同进化框架（ACE），实现二者在大规模决策中的协同优化。ACE引入双角色轨迹精炼机制：训练过程中，大模型同时作为策略执行者（Actor）和价值评判者（Critic）。Actor通过多步推理与环境验证优化次优动作，Critic则通过轨迹级奖励重构完成时间信用分配。与此同时，强化学习代理利用优先经验回放生成高质量数据集，对大模型进行任务定制化微调。在多个电力系统运行挑战中进行的广泛实验表明，该框架在动作空间超过60,000离散动作的条件下，性能显著优于现有强化学习方法及基于大模型的方法。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) and Reinforcement Learning (RL) have shown significant promise in decision-making tasks. Nevertheless, for large-scale industrial decision problems, both approaches face distinct challenges: LLMs lack real-time long-sequence decision-making capabilities, while RL struggles with sample efficiency in vast action spaces. To bridge this gap, we propose Agents Co-Evolution (ACE), a synergistic framework between LLMs and RL agents for large-scale decision-making scenarios. ACE introduces a dual-role trajectory refinement mechanism where LLMs act as both Policy Actor and Value Critic during RL's training: the Actor refines suboptimal actions via multi-step reasoning and environment validation, while the Critic performs temporal credit assignment through trajectory-level reward shaping. Concurrently, RL agent enhances LLMs' task-specific decision-making with high-quality fine-tuning datasets generated via prioritized experience replay. Through extensive experiments across multiple power grid operation challenges with action spaces exceeding 60K discrete actions, ACE demonstrates superior performance over existing RL methods and LLM-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。