测试大模型代理能否通过交互发现隐藏的自动机规则
Can LLM Agents Infer World Models? Evidence from Agentic Automata Learning
- 让大模型通过提问探索未知自动机，模拟智能体推理过程
- 模型性能随自动机规模增大急剧下降，远不如传统算法
- 适合研究大模型推理与规划能力的局限性
我们提出代理型自动机学习方法，评估工具调用的大语言模型代理通过交互揭示隐藏环境的能力。在该设置中，代理需通过两种查询与预言机互动：(1) 成员查询（该字符串是否属于目标语言？）和 (2) 等价查询（这个自动机是否为目标自动机？）。该方法构建了一个可扩展的测试平台，具备可控的任务复杂度、可测量的交互效率以及强基线（经典自动机学习算法）。对当前最先进的大语言模型进行评估发现，随着自动机规模增大，性能显著下降。推理类模型明显优于非推理类模型，但轨迹分析揭示其在查询规划、证据整合和假设构建方面存在反复失败。总体而言，当前大语言模型代理虽能偶尔实现非平凡的交互式发现，但在鲁棒性和效率上仍远逊于经典算法。
原文摘要 · Abstract (English)
We propose agentic automata learning to evaluate the extent to which tool-calling LLM agents can uncover hidden environments through interaction. In our setup, an agent should uncover a hidden deterministic finite automaton (DFA) by interacting with an oracle through (1) membership queries ("Does this string belong to the target language?") and (2) equivalence queries ("Is this the target DFA?"). This yields a scalable testbed with controlled task complexity, measurable interaction efficiency, and strong baselines (classic automata-learning algorithms). Evaluating state-of-the-art LLMs, we find that performance drops sharply as DFA size increases. Reasoning models are markedly stronger than non-reasoning models, yet trajectory analyses reveal recurring failures in query planning, evidence integration, and hypothesis construction. Overall, our results show that current LLM agents can sometimes perform non-trivial interactive discovery, but remain far less robust and efficient than classic algorithms for the task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。