评测大模型处理工具输出的能力，发现即使顶尖模型也难搞定复杂JSON。
How Good Are LLMs at Processing Tool Outputs?
- 构建数据集，测试15个模型对工具返回的JSON数据处理能力。
- 不同处理策略下性能差异达3%至50%，模型表现受输出大小和推理复杂度影响。
- 提示工程需根据输出结构和任务难度定制，通用方法效果差。
大多数现实中的任务自动化问题需要大语言模型（LLMs）调用工具，而工具常返回复杂的JSON响应。这些响应需进一步处理以提取任务完成所需信息，但该能力尚未得到充分研究。本文针对工具响应处理任务，评估了大模型处理结构化（JSON）响应的能力。我们构建了一个专用数据集，并使用多种提示策略对15个开源与闭源模型进行了评估。结果表明，即使在前沿模型中，JSON处理仍是一项困难任务，且在不同提示策略下表现差异显著。最优处理策略取决于工具输出的性质与规模，以及所需推理的复杂度；不同处理方式导致的性能差异范围为3%至50%。
原文摘要 · Abstract (English)
Most realistic task automation problems require large language models (LLMs) to call tools, which often return complex JSON responses. These responses must be further processed to derive the information necessary for task completion. The ability of LLMs to do so is under-studied. In this paper, we study the tool response processing task and LLMs' abilities to process structured (JSON) responses. We created a dataset for this task, and evaluated 15 open and closed weight models using multiple prompting approaches. Our results show that JSON processing remains a difficult task even for frontier models across multiple prompting strategies. The optimal response processing strategy depends on both the nature and size of the tool outputs, as well as the complexity of the required reasoning. Variations in processing approaches can lead to performance differences ranging from 3\% to 50\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。