提升大模型智能体跨领域泛化能力,关键在环境状态信息丰富度。
Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agents
- 通过增加状态中的无关干扰特征,低成本增强状态信息丰富度。
- 状态信息越丰富,智能体在未知领域的泛化性能越强,超越真实场景依赖。
- 逐步思考机制有助于保持跨域泛化,避免灾难性遗忘。
通用大模型智能体通常在有限环境上进行后训练,但部署时需应对更广泛、未见过的领域。本文研究了在测试领域未知情况下,强化学习(RL)训练对智能体泛化的影响。发现两个关键环境维度决定跨域表现:(i) 状态信息丰富度——即智能体需处理的状态信息量;(ii) 规划复杂度——由基础策略下的目标可达性和轨迹长度估计。值得注意的是,领域真实度和文本相似性并非主要因素;例如,简单的方块推箱子(Sokoban)环境在SciWorld中比更真实的ALFWorld带来更强泛化能力。基于此,提出一种低开销随机化方法:在状态中添加少量与目标无关的干扰特征,以提升信息丰富度而不改变任务。此外,实验表明:(a) SFT预热或中期训练虽能防止灾难性遗忘,但会削弱对未包含在数据混合中的新领域的泛化;(b) 在强化学习中开启逐步思考,虽不总提升域内性能,却对维持跨域泛化至关重要。
原文摘要 · Abstract (English)
Generalist LLM agents are often post-trained on a narrow set of environments but deployed across far broader, unseen domains. In this work, we investigate the challenge of agentic post-training when the eventual test domains are unknown. Specifically, we analyze which properties of reinforcement learning (RL) environments and modeling choices have the greatest influence on out-of-domain performance. First, we identify two environment axes that strongly correlate with cross-domain generalization: (i) state information richness, i.e., the amount of information for the agent to process from the state, and (ii) planning complexity, estimated via goal reachability and trajectory length under a base policy. Notably, domain realism and text-level similarity are not the primary factors; for instance, the simple grid-world domain Sokoban leads to even stronger generalization in SciWorld than the more realistic ALFWorld. Motivated by these findings, we further show that increasing state information richness alone can already effectively improve cross-domain robustness. We propose a randomization technique, which is low-overhead and broadly applicable: add small amounts of distractive goal-irrelevant features to the state to make it richer without altering the task. Beyond environment-side properties, we also examine several modeling choices: (a) SFT warmup or mid-training helps prevent catastrophic forgetting during RL but undermines generalization to domains that are not included in the mid-training datamix; and (b) turning on step-by-step thinking during RL, while not always improving in-domain performance, plays a crucial role in preserving generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。