用上下文与训练算力建模任务表现，提升长文本大模型设计效率
Predicting Task Performance with Context-aware Scaling Laws
- 构建上下文与训练算力联合建模框架，预测下游任务表现
- 在6.5万实例上验证，跨三阶算力和上下文长度准确预测性能
- 适用于长文本任务优化，帮助设计更高效的LLM
缩放定律已改变我们对大语言模型的理解，将上游指标如交叉熵损失与模型规模、训练数据量和计算量等设计因素关联起来。然而，这些传统定律无法捕捉下游任务表现，而上下文在此中起关键作用。本文提出一种简单且可解释的框架，联合建模下游性能与训练算力及提供上下文的关系。我们通过在扩展上下文版本的Llama-2-7B和Llama-2-13B上拟合该框架，对65,500个唯一实例（涵盖算术推理、常识推理和机器翻译三类任务）进行实证验证。结果表明，该框架能准确建模分布内下游性能，跨三个数量级的训练算力保持泛化能力，并可靠外推随上下文增加的性能变化。这些发现揭示了训练算力与上下文利用之间的相互作用，为设计适用于多样下游任务的高效长上下文大模型提供了指导。代码已开源：https://github.com/wang-research-lab/context-scaling。
原文摘要 · Abstract (English)
Scaling laws have transformed our understanding of large language models by linking upstream metrics like cross-entropy loss to design factors such as model size, training data, and compute. However, these conventional laws fail to capture downstream task performance, where context plays a critical role. In this work, we propose a straightforward, interpretable framework that jointly models downstream performance as a function of the training compute and the provided context. We empirically validate our framework by fitting it on the observed downstream performance of extended-context variants of Llama-2-7B and Llama-2-13B across 65,500 unique instances spanning three tasks: arithmetic reasoning, common sense reasoning, and machine translation. Our results demonstrate that our framework accurately models in-distribution downstream performance, generalizes across three orders of magnitude in training compute, and reliably extrapolates performance as the amount of context increases. These findings offer valuable insights into the interplay between training compute and context utilization, providing guidance for designing more efficient long-context LLMs for diverse downstream tasks. Our code is available at https://github.com/wang-research-lab/context-scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。