筛选长文本中真正依赖远距离信息的数据，提升长上下文模型训练效率。
Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
- 通过对比长短上下文预测差异，量化文本中远距离依赖的信息量。
- 在8K扩展至64K上下文时，显著提升LLaMA-3-8B在多个基准上的表现。
- 适合需要高效训练长上下文大模型的研究者和工程团队。
长上下文语言模型通过利用长文本跨度中的依赖关系，在推理、代码生成和文档摘要等任务中展现出强大能力。然而，大量可用的长文本数据缺乏有意义的远距离依赖，多数内容仅靠局部上下文即可预测。在这些数据上训练效率低下，因此精心选择训练数据至关重要。为此，我们提出LongFilter框架，用于为长上下文预训练定制高质量数据。LongFilter通过对比长上下文与短上下文设置下的模型预测，衡量扩展上下文带来的信息增益，从而识别出必须依赖远距离依赖的样本。在将LLaMA-3-8B的上下文长度从8K扩展至64K的实验中，LongFilter能高效筛选高质量数据，并在HELMET、LongBench和RULER等多个基准上带来显著提升。
原文摘要 · Abstract (English)
Long-context language models unlock advanced capabilities in reasoning, code generation, and document summarization by leveraging dependencies across extended spans of text. However, a significant portion of readily available long-text data lacks meaningful long-distance dependencies; most spans can be predicted using only local context. Training on such data is inefficient, making careful data selection crucial. Therefore, we introduce LongFilter, a framework for curating training data tailored to long-context pretraining. LongFilter measures the information gain provided by extended context by contrasting model predictions under long-context versus short-context settings, thereby identifying samples where long-range dependencies are essential. Experiments with LLaMA-3-8B, extending its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。