用注意力机制筛选长文本数据，提升大模型长程依赖学习效率
LongAttn: Selecting Long-context Training Data via Token-level Attention
- 基于令牌级注意力计算依赖强度与分布均匀性，精准衡量长距离依赖
- 从开源数据中筛选出32K tokens的高质量长文本数据LongABC-32K
- 方法高效可扩展，适合长文本生成与训练数据构建的研究者使用
随着大语言模型的发展，处理长上下文的需求日益增长。为增强模型的长程依赖建模能力，构建高质量的长上下文训练数据至关重要。现有方法多依赖句级分析，性能与效率均有优化空间。本文提出一种全新的令牌级框架LongAttn，利用大语言模型的自注意力机制量化数据中的长距离依赖关系。通过计算令牌级依赖强度与得分分布均匀性，实现对长程依赖的精准度量，从而更准确、高效地进行数据筛选。我们从开源长上下文数据集（ArXiv、Book、Code）中筛选出LongABC-32K数据集。实验表明，LongAttn在有效性、可扩展性和效率方面表现优异。为推动后续研究，我们公开了代码与高质量长上下文训练数据LongABC-32K。
原文摘要 · Abstract (English)
With the development of large language models (LLMs), there has been an increasing need for significant advancements in handling long contexts. To enhance long-context capabilities, constructing high-quality training data with long-range dependencies is crucial. Existing methods to select long-context data often rely on sentence-level analysis, which can be greatly optimized in both performance and efficiency. In this paper, we propose a novel token-level framework, LongAttn, which leverages the self-attention mechanism of LLMs to measure the long-range dependencies for the data. By calculating token-level dependency strength and distribution uniformity of token scores, LongAttn effectively quantifies long-range dependencies, enabling more accurate and efficient data selection. We filter LongABC-32K from open-source long-context datasets (ArXiv, Book, and Code). Through our comprehensive experiments, LongAttn has demonstrated its excellent effectiveness, scalability, and efficiency. To facilitate future research in long-context data, we released our code and the high-quality long-context training data LongABC-32K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。