arXiv:2608.14109cs.AIcs.LG2026-08

用图结构小模型实时诊断并修复大模型的运行漂移问题。

A Graph-Based Reinforcement Learning Framework for Structured Drift Diagnosis and Recovery in Autonomous LLM Agents

论文配图:A Graph-Based Reinforcement Learning Framework for Structured Drift Diagnosis and Recovery in Autonomous LLM Agents
图 1 · 摘自论文原文
  • 构建图结构框架,让小模型在每个节点专注不同诊断任务。
  • 在AppWorld上实现90%以上正确恢复决策,且输出格式规范。
  • 适合部署在无法重训练的大模型系统中做即插即用的故障恢复。

自主大模型代理在复杂真实工作流中应用日益广泛,但易发生运行时行为漂移,这种隐性偏差可能对外部系统造成不可逆影响。现有方法仅在提示层处理漂移,缺乏步骤级检测、风险评估与恢复决策的结构化机制。由于主任务代理通常是难以重训练的大模型,本文提出一种即插即用的恢复模块:基于图结构,用强化学习训练一个小型语言模型,在图的每个节点上分别承担漂移分类、操作识别、风险评估或最终决策角色,生成符合特定角色的结构化XML推理。训练结合规则化的结构奖励与大模型作为裁判的语义质量信号,从格式和内容两方面评估模型表现。在公开的AppWorld基准测试中,该方法能有效利用漂移发生前的线索做出正确恢复决策;同时,小模型可靠遵循指定输出格式,并在各节点生成语义恰当的内容。

原文摘要 · Abstract (English)

Autonomous LLM agents are increasingly deployed in complex real-world workflows, yet they remain vulnerable to runtime behavioral drift, a silent deviation from the original task that can lead to irreversible side effects on external systems. Existing approaches address drift at the prompt level but lack structured mechanisms for step-level detection, risk assessment, and recovery decision. Because the main task-executing agent is often a large and expensive model that cannot be re-trained on every deployment, this work targets a plug-and-play recovery module instead. It introduces a graph-based framework in which a single small language model is trained via reinforcement learning to specialize at each node of a recovery graph, external to the main agent. Each node has a precise role\,: drift classification, operation detection, risk evaluation, or final decision and the model learns to produce structured XML-formatted reasoning adapted to that role. Training combines rule-based structural rewards with an LLM-as-judge semantic-quality signal, so that the model is graded both on how it answers (schema and length) and on what it says. Experiments on the public AppWorld benchmark show that the method generally exploits information about the suspected drift onset to issue correct recovery decisions using a small language model. In addition, the trained small language model reliably respects the prescribed output schema and produces semantically appropriate content in each field according to its assigned node role.

大模型故障恢复强化学习图神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。