用掩码扩散模型构建可调控的文本世界模型，提升强化学习环境多样性与泛化能力。
Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
- 提出双向锚定去噪的掩码扩散语言模型，突破自回归模型的单向依赖限制。
- 在9个开源环境、12个模型上实现47%零样本迁移性能提升，参数量为基线4倍仍保持低延迟。
- 支持工具架构、任务上下文等全局条件控制，适合构建可控、可扩展的智能体训练环境。
近年来强化学习的发展催生了对多样化、专业化训练环境的需求。传统手工设计的固定任务与奖励环境在模型性能提升后失效，长时程稀疏奖励易引发特定工作流或工具结构的模式坍塌。基于状态模拟的世界模型已达到纯滚动回放的性能水平，具备按需扩展多样性的潜力。然而自回归语言模型存在从左到右的偏见，难以对全局相关状态锚点（如工具结构、历史对话、预期结果）进行有效条件建模。本文将文本世界建模形式化为可调控的动态转移问题，分解为初始状态、任务上下文、工具结构、领域规则与调控指令，并构建了包含239,403条真实状态-动作轨迹的数据集，覆盖九个开源环境与十二个前沿模型家族。对比自回归语言模型与掩码扩散语言模型（MDLMs），发现后者通过双向锚定去噪，在参数量为基线4倍的情况下，仍保持相当的推理延迟，且在连贯性、真实性与滚动回放多样性上表现更优。我们引入即插即用的GRPO训练框架，结合确定性状态校验，并在三个域外环境（ScienceWorld、ALFWorld、AppWorld）上对三类1.2B–7B规模智能体（LFM2.5、Qwen3、Mistral）进行零样本迁移实验，性能相比基线最高提升47%，无需环境微调。进一步开展对抗场景下的行为分析与人工评估，验证其现实性、结果正确性与训练价值。代码已开源，以推动该方向研究。
原文摘要 · Abstract (English)
Recent growth in reinforcement learning (RL) has surfaced a need for diverse, specialized training environments. Hand-curated environments with fixed task and reward difficulties become ineffective signals as model performance improves, and sparse rewards over long horizons induce mode collapse on specific workflows or tool structures. World models that simulate environment states have matched pure rollout performance, making them promising for scaling diversity on-demand. However, autoregressive (AR) world models suffer from a left-to-right bias preventing conditioning on globally interdependent state anchors such as tool schemas, prior turns, and expected outcomes. We (i) formalize text-based world modeling as a steerable transition-dynamics problem decomposed into initial state, task context, tool schemas, domain rules, and steering directives, and (ii) curate 239,403 grounded state-action trajectories spanning nine open-source environments and twelve frontier model families. We compare AR LMs and masked diffusion language models (MDLMs), showing MDLMs, via bidirectional anchor-aware denoising, achieve better coherence, groundedness, and empirically validated rollout diversity than LLMs over 4x their parameter size, at comparable inference latency. We introduce a plug-and-play GRPO training framework with deterministic state checks, and perform zero-shot transfer ablations on three OOD environments (ScienceWorld, ALFWorld, AppWorld) across three 1.2B-7B agent backbones (LFM2.5, Qwen3, Mistral), achieving up to 47% absolute gains over baselines without environment-specific fine-tuning. We further conduct behavioral analysis of failure modes under adversarial scenarios and human evaluation on realism, outcome correctness, and training utility. We open-source our work to encourage research in this direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。