测试大模型能否根据环境状态判断动作是否可行，发现现有模型表现差。
GroundAct: Can LLM Agents Ground Actions in Environmental States?
- 构建1500个场景的基准测试，评估模型对环境状态的动作推理能力。
- 模型在显式指令下成功率85%-96%，隐含状态依赖任务仅29%-53%。
- 揭示模型在工具使用与协作推理上的能力差异，适合研究智能体推理瓶颈者关注。
当指令完整描述动作时，大模型智能体的成功率可达85%-96%，但当动作可行性依赖指令未提及的环境状态时，成功率骤降至29%-53%。我们提出这一差距源于‘动作接地’能力的缺失：即从结构化环境状态中推断动作可行性、前置条件缺失情况及个体能力极限。为此，我们引入GroundAct，一个包含1,500个场景和16,592个任务实例的基准，覆盖11个领域，任务按认知复杂度分为七类。评估15个大模型（3B-671B），发现三个诊断模式：(i) 属性推理与工具/协作推理弱相关，形成不同模型特征；(ii) 完整环境图可使工具使用性能提升+27.6%，协作隐含时下降-22.9%，区分搜索瓶颈与约束过滤瓶颈；(iii) 监督微调使Qwen2.5-3B在直接命令任务上从0.6%提升至76.3%，但在隐含协作任务上仅从1.5%升至5.5%。结果表明动作接地是多维挑战，不可通过单纯缩放解决。
原文摘要 · Abstract (English)
LLM agents achieve 85-96% success on tasks where instructions fully specify the action, but drop to 29-53% when action feasibility depends on environmental state that the instruction does not mention. We argue that this gap reflects a missing capability: action grounding, the ability to infer from structured environmental state whether an action is feasible, what prerequisites it lacks, and whether it exceeds individual capacity. We introduce GroundAct, a benchmark of 1,500 scenarios and 16,592 task instances in text-based interactive environments spanning 11 domains, with tasks organized into seven categories along a cognitive complexity hierarchy. Evaluating 15 LLMs (3B-671B), we find three diagnostic patterns: (i) attribute reasoning is weakly correlated with tool and coordination reasoning, producing distinct model profiles; (ii) complete environment graphs yield up to +27.6/-22.9% on tool use vs. implicit collaboration, separating search-bound from constraint-filtering bottlenecks; and (iii) supervised fine-tuning lifts Qwen2.5-3B from 0.6% to 76.3% on direct command but only 1.5% to 5.5% on implicit collaboration. These results establish action grounding as a multi-dimensional challenge irreducible to scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。