用语言元数据增强嵌入，让大模型更高效地预训练。
LIME: Making LLM Data More Efficient with Linguistic Metadata Embeddings
- 将语法语义等元数据融入词元嵌入，提升训练效率。
- 训练速度加快56%，参数仅增0.01%，计算开销几乎不变。
- 适用于不同规模模型，尤其适合需要强生成能力的场景。
预训练解码器仅语言模型依赖大量高质量数据，但此类数据日益逼近极限。尽管元数据常用于数据集构建与筛选，其作为直接训练信号的潜力仍被低估。本文提出LIME（Linguistic Metadata Embeddings），通过在词元嵌入中融合捕捉句法、语义和上下文特性的元数据，显著提升预训练效率：相比基线模型，可加速56%适应数据分布，额外参数仅增加0.01%，计算开销几乎无增长。此外，LIME改善了分词效果，带来更强的语言建模能力和生成任务表现，且在500M至2B模型规模上均有效。我们还提出变体LIME+1，利用前一词的元数据引导下一词生成，使推理性能最高提升38%，算术准确率最高提升35%。
原文摘要 · Abstract (English)
Pre-training decoder-only language models relies on vast amounts of high-quality data, yet the availability of such data is increasingly reaching its limits. While metadata is commonly used to create and curate these datasets, its potential as a direct training signal remains under-explored. We challenge this status quo and propose LIME (Linguistic Metadata Embeddings), a method that enriches token embeddings with metadata capturing syntax, semantics, and contextual properties. LIME substantially improves pre-training efficiency. Specifically, it adapts up to 56% faster to the training data distribution, while introducing only 0.01% additional parameters at negligible compute overhead. Beyond efficiency, LIME improves tokenization, leading to remarkably stronger language modeling capabilities and generative task performance. These benefits persist across model scales (500M to 2B). In addition, we develop a variant with shifted metadata, LIME+1, that can guide token generation. Given prior metadata for the next token, LIME+1 improves reasoning performance by up to 38% and arithmetic accuracy by up to 35%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。