让智能体学会复用通用技能，提升泛化能力
Skill Reuse as Compression in Agentic RL
- 用最小描述长度原则压缩成功轨迹，提取可复用的抽象技能
- 在三个环境中均优于基线，在分布外任务上成功率提升15%以上
- 适合研究智能体泛化与高效学习的学者
基于强化学习训练的大语言模型智能体常习得脆弱且任务特定的捷径。我们假设：当成功轨迹具有结构可压缩性，能分解为少量可复用的抽象模式时，智能体泛化能力更强。为此，提出ReuseRL，将智能体强化学习建立在最小描述长度（MDL）原则之上。ReuseRL从成功轨迹中提取共享技能字典，并在强化学习目标中引入分割代价，显式惩罚编码不佳的个性行为。我们证明了一个PAC-Bayes界，保证从成功轨迹中提取的字典在未来成功行为上的期望描述长度有界。在ALFWorld、TextWorld-Cooking和Countdown-Stepwise三个环境中，ReuseRL在分布内和分布外任务上的成功率均优于原始GRPO和强基线方法。
原文摘要 · Abstract (English)
Large language model agents trained with reinforcement learning (RL) often learn brittle, task-specific shortcuts. We hypothesize that agents generalize better when their successful trajectories are structurally compressible, decomposed into a small set of reusable abstract patterns. To formalize this, we introduce ReuseRL, which grounds agentic RL in the Minimum Description Length (MDL) principle. ReuseRL extracts a shared skill dictionary from successful trajectories and augments the RL objective with a segmentation cost, explicitly penalizing idiosyncratic behaviors that encode poorly. We prove a PAC-Bayes bound guaranteeing that a dictionary extracted from successful trajectories has bounded expected description length on future successful behavior. Across ALFWorld, TextWorld-Cooking, and Countdown-Stepwise, ReuseRL improves in- and out-of-distribution success over vanilla GRPO and strong round-length baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。