测试大模型追踪多个实体状态的能力，发现新模型结合思维链更可靠。
Exploring State Tracking Capabilities of Large Language Models
- 设计三个明确的状态追踪任务构建基准测试
- GPT-4和Llama3能持续追踪状态，旧模型多在后期失效
- 适合研究模型长期推理与状态记忆的学者参考
大型语言模型（LLMs）在解决复杂任务方面展现出强大能力，包括需要一定推理水平的任务。本文聚焦于状态追踪问题，即模型需持续跟踪多个实体所处状态。为隔离状态追踪这一核心能力，我们基于三个定义清晰的任务构建了基准测试，并分析了不同场景下LLMs的表现。结果表明，最新一代模型（特别是GPT-4和Llama3）具备状态追踪能力，尤其在结合思维链（Chain of Thought）机制时表现更优。而前代模型虽能理解任务并在初期解决，但在经过一定步骤后通常会失败。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive capabilities in solving complex tasks, including those requiring a certain level of reasoning. In this paper, we focus on state tracking, a problem where models need to keep track of the state governing a number of entities. To isolate the state tracking component from other factors, we propose a benchmark based on three well-defined state tracking tasks and analyse the performance of LLMs in different scenarios. The results indicate that the recent generation of LLMs (specifically, GPT-4 and Llama3) are capable of tracking state, especially when integrated with mechanisms such as Chain of Thought. However, models from the former generation, while understanding the task and being able to solve it at the initial stages, often fail at this task after a certain number of steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。