用黑箱环境测试大模型的交互式推理能力，发现其规划策略仍不足。
Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interaction
- 通过与隐藏函数互动来评估模型推理，模拟真实探索过程。
- 19个模型在96个环境中测试，o3在5项任务中领先，但难题平均准确率低于40%。
- 揭示大模型缺乏高效自适应探索策略，适合研究智能体与推理的学者参考。
现有任务难以评估大语言模型在未知交互环境中的推理能力，导致对演绎、归纳和溯因推理的孤立评估，忽视了人类式发现学习所必需的综合推理过程。本文提出一种新型评估范式——黑箱环境交互，其中黑箱环境由隐藏函数定义，模型需通过有限次交互探索，从输入输出对中推断隐藏函数。基于此构建了包含6类任务、96个黑箱环境的 extsc{Oracle}基准，评估19个主流大模型。OpenAI的o3在6项任务中5项排名第一，在多数简单任务上准确率超70%，但在部分难题上平均表现低于40%。进一步分析显示，所有模型普遍存在高阶规划缺陷，无法制定高效且自适应的探索策略以优化假设。代码已开源。
原文摘要 · Abstract (English)
Existing tasks fall short in evaluating reasoning ability of Large Language Models (LLMs) in an interactive, unknown environment. This deficiency leads to the isolated assessment of deductive, inductive, and abductive reasoning, neglecting the integrated reasoning process that is indispensable for human-like discovery learning. We introduce a novel evaluation paradigm, \textit{black-box environment interaction}, to tackle this challenge. A black-box environment is defined by a hidden function that maps a specific set of inputs to outputs. LLMs are required to unravel the hidden function behind the black-box environment by interacting with it in given exploration turns, and reasoning over observed input-output pairs. Leveraging this idea, we build the \textsc{Oracle} benchmark which comprises 6 types of black-box task with 96 black-box environments. 19 modern LLMs are benchmarked. o3, a leading LLM from OpenAI, ranks first in 5 of the 6 tasks, achieving over 70\% accuracy on most easy black-box environments. But it still struggles with some hard black-box tasks, where the average performance drops below 40\%. Further analysis reveals a universal difficulty among LLMs: They lack the high-level planning capability to develop efficient and adaptive exploration strategies for hypothesis refinement. Code is available in https://github.com/lemonsis/Oracle_Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。