构建真实动态工具使用环境,测试大模型应对时空突变的重规划能力。
STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics

- 设计227个带时空干扰的交互任务,模拟现实计划中断场景。
- 顶尖模型准确率不足40%,暴露其在动态推理中的根本缺陷。
- 提出迭代优化与在线强化学习结合的新方法,显著提升适应性。
在真实代理应用中,大语言模型需在任务执行过程中因突发变化而重新规划与调整。现有动态基准主要评估模型对时间变化的检测能力,却忽视了在时空动态下进行自适应重规划的关键挑战。本文提出STT-Arena(时空工具使用基准),包含227个高质量交互任务,覆盖九类时空冲突类型和四类可解性等级。每个任务基于可执行的真实环境,嵌入时空触发器,可突然使原计划失效,迫使模型识别状态变化并制定新策略。对前沿大模型的广泛评估显示,即使是最先进的专有模型(如Claude-4.6-Opus)整体准确率也低于40%,凸显时空动态推理的困难。系统性分析失败轨迹发现三类常见错误:过时状态执行、动态触发误判、适应后验证缺失。基于此,我们提出一种迭代轨迹精炼技术,从训练数据中消除这些错误模式,并结合在线强化学习,构建出STT-Agent-4B,在STT-Arena上优于当前领先模型。
原文摘要 · Abstract (English)
Large language models (LLMs) deployed in real-world agentic applications must be capable of replanning and adapting when mid-task disruptions invalidate their prior decisions. Existing dynamic benchmarks primarily measure whether LLMs can detect temporal changes in a timely manner, leaving the complementary challenge of adaptive replanning under spatio-temporal dynamics largely unexplored. We introduce STT-Arena (Spatio-Temporal Tool-Use Arena), a benchmark of 227 high-quality interactive tasks spanning nine spatio-temporal conflict types and four solvability levels. Each task is grounded in a realistic, executable environment equipped with injected spatio-temporal triggers that can abruptly invalidate an ongoing plan, forcing the model to detect the state shift and construct a revised execution strategy. Extensive evaluation of frontier LLMs reveals that even the SOTA proprietary models, including Claude-4.6-Opus, achieves less than 40\% overall accuracies, highlighting the fundamental difficulty of spatio-temporal dynamic reasoning. Systematic analysis of failure trajectories uncovers three recurring error modes of existing models: Stale-State Execution, Misdiagnosis of Dynamic Triggers, and Missing Post-Adaptation Verification. Guided by these findings, we propose an iterative trajectory refinement technique that eliminates these failure patterns from training data, and combine it with online RL to produce STT-Agent-4B which outperforms frontier LLMs on STT-Arena.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。