arXiv:2509.03990cs.AI2025-09被引 7

让大模型智能体学会复用反思经验,提升跨任务表现。

Meta-Policy Reflexion: Reusable Reflective Memory and Rule Admissibility for Resource-Efficient LLM Agent

  • 将反思内容结构化存入可复用的记忆库,无需更新模型参数。
  • 在推理时通过规则检查减少无效操作,准确率显著提升。
  • 适合需要长期学习和安全执行的复杂任务场景。

大型语言模型(LLM)智能体在单任务上表现优异,但常出现重复失败、探索效率低及跨任务适应性差的问题。现有反思策略(如Reflexion、ReAct)生成的反思痕迹通常短暂且仅限单一任务使用;基于强化学习的替代方案虽具迁移性,却需大量参数更新与计算资源。本文提出元策略反思(Meta-Policy Reflexion, MPR):一种混合框架,将LLM生成的反思内容归纳为结构化的谓词式元策略记忆(MPM),并在推理阶段通过软记忆引导解码与硬规则可接受性检查(HAC)两种机制应用。MPR(i)在不更新模型权重的情况下外化可复用的修正知识,(ii)通过领域约束减少不安全或无效动作,(iii)保持语言型反思的灵活性。我们形式化了MPM表示,提出了更新与解码算法,并在基于AlfWorld的文本代理环境中验证该方法。实证结果表明,相比Reflexion基线,MPR在执行准确率与鲁棒性上均有持续提升;规则可接受性进一步增强稳定性。我们分析了性能提升机制,讨论了可扩展性与失败模式,并展望了多模态与多智能体方向的未来工作。

原文摘要 · Abstract (English)

Large language model (LLM) agents achieve impressive single-task performance but commonly exhibit repeated failures, inefficient exploration, and limited cross-task adaptability. Existing reflective strategies (e.g., Reflexion, ReAct) improve per-episode behavior but typically produce ephemeral, task-specific traces that are not reused across tasks. Reinforcement-learning based alternatives can produce transferable policies but require substantial parameter updates and compute. In this work we introduce Meta-Policy Reflexion (MPR): a hybrid framework that consolidates LLM-generated reflections into a structured, predicate-like Meta-Policy Memory (MPM) and applies that memory at inference time through two complementary mechanisms soft memory-guided decoding and hard rule admissibility checks(HAC). MPR (i) externalizes reusable corrective knowledge without model weight updates, (ii) enforces domain constraints to reduce unsafe or invalid actions, and (iii) retains the adaptability of language-based reflection. We formalize the MPM representation, present algorithms for update and decoding, and validate the approach in a text-based agent environment following the experimental protocol described in the provided implementation (AlfWorld-based). Empirical results reported in the supplied material indicate consistent gains in execution accuracy and robustness when compared to Reflexion baselines; rule admissibility further improves stability. We analyze mechanisms that explain these gains, discuss scalability and failure modes, and outline future directions for multimodal and multi-agent extensions.

大模型智能体反思机制记忆增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。