用知识图谱增强语言模型，提升工业文本嵌入效果。
Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry
- 基于图嵌入构造三元组，进行对比学习
- 在工业文本任务上比现有模型高5.45-7.96个百分点
- 参数量仅为三分之一，适合资源受限场景
近期自然语言处理趋势利用知识图谱（KG）通过图结构引入额外知识，增强预训练语言模型对领域术语或文档间关系的捕捉能力。本文研究将原用于科学文献的图感知邻域对比学习方法SciNCL应用于过程工业领域，该领域文本日志蕴含关键操作信息，且常表现为稀疏知识图谱。实验表明，使用图嵌入生成的三元组微调语言模型，在自研工业文本嵌入基准（PITEB）上相比最优的mE5-large编码器性能提升5.45-7.96个百分点（9.8%-14.3%），同时参数量仅为后者的三分之一。
原文摘要 · Abstract (English)
Recent trends in NLP utilize knowledge graphs (KGs) to enhance pretrained language models by incorporating additional knowledge from the graph structures to learn domain-specific terminology or relationships between documents that might otherwise be overlooked. This paper explores how SciNCL, a graph-aware neighborhood contrastive learning methodology originally designed for scientific publications, can be applied to the process industry domain, where text logs contain crucial information about daily operations and are often structured as sparse KGs. Our experiments demonstrate that language models fine-tuned with triplets derived from graph embeddings (GE) outperform a state-of-the-art mE5-large text encoder by 9.8-14.3% (5.45-7.96p) on the proprietary process industry text embedding benchmark (PITEB) while having 3 times fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。