测试大模型在资源有限时的前瞻性决策能力，发现其表现仍有巨大提升空间。
TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
- 设计框架TRIAGE，让模型提前规划任务选择、顺序和资源分配
- 在数学、科学等多领域测试中，模型效率远低于理想水平
- 适合关注智能体资源调度与高效部署的研究者
将语言模型作为自主代理部署时，仅靠单任务准确率不足：当代理面临一系列问题且拥有有限的令牌预算时，必须在无执行反馈前决定处理哪些问题、以何种顺序处理以及为每个问题投入多少计算资源。这是人类认知中长期研究的前瞻性元认知控制，但语言模型是否具备此能力尚未被检验。我们提出TRIAGE评估框架，模型接收一个任务池和与其基线成本匹配的令牌预算，需提交一份包含任务选择、排序及每题资源分配的单一有序计划。计划评分基于一个已知模型对各问题可解性与成本的全知“最优”策略，生成统一尺度的分诊效率比。我们在竞赛数学、研究生级科学、代码生成及专家多学科知识等领域，评估前沿与开源模型（含/不含推理能力），发现当前语言模型在前瞻性元认知控制上存在显著差距，揭示了一个此前未测量的能力维度，对资源高效的代理部署具有直接意义。
原文摘要 · Abstract (English)
Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per-problem allocation. Plans are scored against an oracle with full knowledge of the model's solvability and cost on each problem, yielding a triage efficiency ratio on a common scale. We evaluate frontier and open-source models, with and without reasoning enabled, across competition mathematics, graduate-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource-efficient agent deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。