让搜索智能体通过自我反思跨回合优化策略，提升测试时探索效率。
Meta-Reinforcement Learning with Self-Reflection for Agentic Search
- 基于自反思机制，利用过往回合经验动态调整搜索策略。
- 在8个基准上实现9.2%至19.3%的性能提升，显著优于基线方法。
- 适合需要持续学习与适应性搜索的智能体系统，如复杂任务规划。
本文提出MR-Search，一种用于智能体搜索的上下文内元强化学习（meta-RL）框架，引入自我反思机制。不同于单次独立回合中稀疏奖励下的策略优化，MR-Search训练一个依赖历史回合的策略，使其能在多回合间自适应调整搜索行为。该方法通过每回合后生成显式自我反思，并将其作为额外上下文指导后续尝试，从而提升测试时的探索效率。我们进一步设计了一种多轮强化学习算法，在回合层级估计密集相对优势，实现对每一轮的精细信用分配。在多个基准上的实验表明，相比基线强化学习方法，MR-Search在8个任务上取得9.2%至19.3%的相对性能提升，展现出强泛化能力。代码与数据已开源于https://github.com/tengxiao1/MR-Search。
原文摘要 · Abstract (English)
This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。