研究提示复杂度对大模型上下文学习能力的限制，发现即使有示例也难突破性能瓶颈。
Assessing the Limits of In-Context Learning beyond Functions using Partially Ordered Relation
- 通过逐步增加提示复杂度，测试大模型在部分有序关系上的上下文学习能力。
- 随着提示复杂度提升，模型性能趋于饱和，表明其学习能力存在隐性上限。
- 适用于关注大模型推理边界与提示工程优化的研究者。
无需更新模型参数，大型语言模型（LLM）通过示例演示即可生成合理且普遍准确的响应，展现出强大的上下文学习（ICL）能力。尽管已有大量研究聚焦于文档级概念的推断，但对上下文中学习除了函数外更明确关系的行为仍需深入探讨。本文通过引入提示中归纳递增的复杂性，评估了大模型在部分有序关系上的表现。实证结果表明，即便提供充足示例，随着提示复杂度增加，所选指标的性能仍趋于饱和，说明上下文学习的效果受限。这一现象在理论上亦可通过其隐式优化过程得到解释。代码已公开。
原文摘要 · Abstract (English)
Generating rational and generally accurate responses to tasks, often accompanied by example demonstrations, highlights Large Language Model's (LLM's) remarkable In-Context Learning (ICL) capabilities without requiring updates to the model's parameter space. Despite having an ongoing exploration focused on the inference from a document-level concept, its behavior in learning well-defined functions or relations in context needs a careful investigation. In this article, we present the performance of ICL on partially ordered relation by introducing the notion of inductively increasing complexity in prompts. In most cases, the saturated performance of the chosen metric indicates that while ICL offers some benefits, its effectiveness remains constrained as we increase the complexity in the prompts even in presence of sufficient demonstrative examples. The behavior is evident from our empirical findings and has further been theoretically justified in term of its implicit optimization process. The code is available \href{https://anonymous.4open.science/r/ICLonPartiallyOrderSet}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。