让大模型一步生成实体，解决知识图谱与文本粒度不匹配问题。
K-ON: Stacking Knowledge On the Head Layer of Large Language Model
- 用多头层预测未来k步，实现实体级一次生成。
- 在知识图谱任务上超越现有文本及多模态方法。
- 支持实体对比损失，适合知识增强型应用。
大型语言模型（LLM）在自然语言处理任务中表现优异，通常通过预测下一个词来训练，但这种设计与知识图谱（KG）的实体单位存在粒度不匹配问题。在知识图谱中，实体需由多个词构成，而传统模型难以一次性识别。为此，本文提出K-ON，通过在模型头部引入多头层，实现对接下来k步的联合预测，从而可一步生成实体。该方法不仅支持实体级输出，还允许使用对比损失进行实体表示学习，显著提升知识图谱建模能力。实验表明，K-ON在多项知识图谱任务上优于当前最优的文本融合方法，甚至超过结合多模态信息的方法，验证了其有效性与泛化能力。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have significantly improved various natural language processing (NLP) tasks. Typically, LLMs are trained to predict the next token, aligning well with many NLP tasks. However, in knowledge graph (KG) scenarios, entities are the fundamental units and identifying an entity requires at least several tokens. This leads to a granularity mismatch between KGs and natural languages. To address this issue, we propose K-ON, which integrates KG knowledge into the LLM by employing multiple head layers for next k-step prediction. K-ON can not only generate entity-level results in one step, but also enables contrastive loss against entities, which is the most powerful tool in KG representation learning. Experimental results show that K-ON outperforms state-of-the-art methods that incorporate text and even the other modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。