给语言模型预训练加入知识坐标，提升上下文理解与生成准确性。
KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates
- 用三维语义坐标表示文档，作为文本前缀参与预训练。
- 在10个下游任务中性能显著提升，训练收敛速度加快约30%。
- 帮助模型区分事实与噪声，有效减少幻觉生成，适合高可靠性场景。
标准大语言模型预训练通常将语料视为扁平的词元序列，忽略了人类理解信息时依赖的真实世界背景。为弥合这一差距，我们提出知识坐标条件化（KoCo），一种将每篇文档映射到三维语义坐标的简单方法。通过在预训练时将这些坐标作为文本前缀添加，旨在赋予模型显式的上下文感知能力，使其在真实世界知识结构中学习文档。实验结果表明，KoCo在10个下游任务中显著提升性能，并使预训练收敛速度加快约30%。进一步分析显示，显式建模知识坐标有助于模型区分稳定事实与噪声，有效缓解生成内容中的幻觉问题。
原文摘要 · Abstract (English)
Standard Large Language Model (LLM) pre-training typically treats corpora as flattened token sequences, often overlooking the real-world context that humans naturally rely on to contextualize information. To bridge this gap, we introduce Knowledge Coordinate Conditioning (KoCo), a simple method that maps every document into a three-dimensional semantic coordinate. By prepending these coordinates as textual prefixes for pre-training, we aim to equip the model with explicit contextual awareness to learn the documents within the real-world knowledge structure. Experiment results demonstrate that KoCo significantly enhances performance across 10 downstream tasks and accelerates pre-training convergence by approximately 30\%. Furthermore, our analysis indicates that explicitly modeling knowledge coordinates helps the model distinguish stable facts from noise, effectively mitigating hallucination in generated outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。