用大模型让强化学习决策变透明,用户能直接问‘为什么’并获得自然语言解释。
TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models
- 设计五类专用大模型代理,自动理解用户问题并匹配解释工具。
- 在四水箱控制任务中准确映射查询,生成可执行的反事实场景。
- 适合需要理解复杂决策过程的研究者与工程师,尤其关注可解释性。
可解释强化学习(XRL)有助于提升强化学习(RL)智能体的透明度。然而,由于现有XRL结果难以理解且方法孤立,领域专家难以选择合适工具。为此,我们提出TalkToAgent,一个基于多智能体大语言模型(LLM)的框架,实现对RL策略的交互式自然语言解释。该框架包含五类专用代理:协调者、解释者、编码器、评估者和调试器,可将用户提问自动映射至相应XRL任务,并从关键状态变量、预期结果或反事实角度阐明智能体行为。此外,我们的方法通过定性行为描述或新规则策略生成替代场景,扩展了传统反事实解释。我们在经典的非线性四水箱过程控制问题上验证了该框架,结果显示其能高精度映射用户查询至XRL任务,编码器-调试器协作显著降低反事实生成失败率。定性评估表明,TalkToAgent有效解释了智能体行为,并将其置于具体问题背景下进行语义定位。
原文摘要 · Abstract (English)
Explainable Reinforcement Learning (XRL) has emerged as a promising approach in improving the transparency of Reinforcement Learning (RL) agents. However, there remains a gap between complex RL policies and domain experts, due to the limited comprehensibility of XRL results and isolated coverage of current XRL approaches that leave users uncertain about which tools to employ. To address these challenges, we introduce TalkToAgent, a multi-agent Large Language Models (LLM) framework that delivers interactive, natural language explanations for RL policies. The architecture with five specialized LLM agents (Coordinator, Explainer, Coder, Evaluator, and Debugger) enables TalkToAgent to automatically map user queries to relevant XRL tools and clarify an agent's actions in terms of either key state variables, expected outcomes, or counterfactual explanations. Moreover, our approach extends previous counterfactual explanations by deriving alternative scenarios from qualitative behavioral descriptions, or even new rule-based policies. We validated TalkToAgent on quadruple-tank process control problem, a well-known nonlinear control benchmark. Results demonstrated that TalkToAgent successfully mapped user queries into XRL tasks with high accuracy, and coder-debugger interactions minimized failures in counterfactual generation. Furthermore, qualitative evaluation confirmed that TalkToAgent effectively interpreted agent's actions and contextualized their meaning within the problem domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。