构建真实场景下的上下文学习基准,揭示大模型在复杂任务中学习新知识的短板。
CL-bench: A Benchmark for Context Learning
- 设计500个真实复杂上下文,要求模型从文本中提取新知识解题
- 10个前沿大模型平均仅解决17.2%的任务,最优模型也仅达23.7%
- 适合研究通用智能、现实场景推理与模型泛化能力的研究者
当前语言模型擅长利用预训练知识进行推理,但真实任务高度依赖上下文:模型需从特定任务上下文中学习新知识以完成推理。我们将其称为上下文学习,这是人类自然具备却长期被忽视的能力。为此,我们提出CL-bench,一个由领域专家构建的真实世界基准,包含500个复杂上下文、1,899个任务和31,607条验证标准。每个任务所需的新内容均存在于对应上下文中。解决这些任务需模型学习领域知识、规则系统、复杂流程或基于数据推导的法律,这些均未出现在预训练中。这远超长上下文检索或简单示例学习任务。对10个前沿语言模型的评估显示,平均仅17.2%的任务被解决,最优模型GPT-5.1也仅达23.7%,表明大模型尚未具备有效上下文学习能力,成为应对真实复杂任务的关键瓶颈。CL-bench为发展此类核心能力提供了重要基础,推动模型向更智能、更实用方向演进。
原文摘要 · Abstract (English)
Current language models (LMs) excel at reasoning over prompts using pre-trained knowledge. However, real-world tasks are far more complex and context-dependent: models must learn from task-specific context and leverage new knowledge beyond what is learned during pre-training to reason and resolve tasks. We term this capability context learning, a crucial ability that humans naturally possess but has been largely overlooked. To this end, we introduce CL-bench, a real-world benchmark consisting of 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics, all crafted by experienced domain experts. Each task is designed such that the new content required to resolve it is contained within the corresponding context. Resolving tasks in CL-bench requires models to learn from the context, ranging from new domain-specific knowledge, rule systems, and complex procedures to laws derived from empirical data, all of which are absent from pre-training. This goes far beyond long-context tasks that primarily test retrieval or reading comprehension, and in-context learning tasks, where models learn simple task patterns via instructions and demonstrations. Our evaluations of ten frontier LMs find that models solve only 17.2% of tasks on average. Even the best-performing model, GPT-5.1, solves only 23.7%, revealing that LMs have yet to achieve effective context learning, which poses a critical bottleneck for tackling real-world, complex context-dependent tasks. CL-bench represents a step towards building LMs with this fundamental capability, making them more intelligent and advancing their deployment in real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。