测试时学习能力可量化，发现大模型进步慢于人类。
How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparison
- 用语义游戏测试模型在测试中边做边学的能力
- 大模型能提升性能但累积经验下进步更慢且不稳定
- 适合关注模型动态学习能力的研究者参考
随着大语言模型评估设计影响人工通用智能的发展路径，全面而前瞻性的评测至关重要。现有基准多衡量静态知识，而智能还包含从经验中快速学习的能力。为此，我们倡导评测测试时学习——即在测试过程中通过经验驱动、推理密集型任务提升表现的能力。本文提出语义游戏作为有效评测平台，因其抗饱和且天然需要策略推理。我们构建了客观评估框架，对比模型在有限与累积经验下的表现，包含四种经验表示形式。为提供参照，招募八名人类参与者完成相同任务。结果表明，大语言模型具备可测量的测试时学习能力；然而，在累积经验下改进更不稳定，进展速度远慢于人类。这凸显了大语言模型作为通用学习机器的潜力，也揭示其与人类间存在显著认知差距，无论其在静态基准上表现如何优异。
原文摘要 · Abstract (English)
As evaluation designs of large language models may shape our trajectory toward artificial general intelligence, comprehensive and forward-looking assessment is essential. Existing benchmarks primarily assess static knowledge, while intelligence also entails the ability to rapidly learn from experience. To this end, we advocate for the evaluation of Test-time Learning, the capacity to improve performance in experience-based, reasoning-intensive tasks during test time. In this work, we propose semantic games as effective testbeds for evaluating test-time learning, due to their resistance to saturation and inherent demand for strategic reasoning. We introduce an objective evaluation framework that compares model performance under both limited and cumulative experience settings, and contains four forms of experience representation. To provide a comparative baseline, we recruit eight human participants to complete the same task. Results show that LLMs exhibit measurable test-time learning capabilities; however, their improvements are less stable under cumulative experience and progress more slowly than those observed in humans. These findings underscore the potential of LLMs as general-purpose learning machines, while also revealing a substantial intellectual gap between models and humans, irrespective of how well LLMs perform on static benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。