用图神经网络增强上下文编码，小资源下实现高效词预测
Next Word Suggestion using Graph Neural Network
- 用图卷积网络建模词语间语义关系，提取上下文特征
- 在有限算力下，基于维基百科数据集实现良好词预测性能
- 适合资源受限场景下的轻量级语言模型设计
自然语言处理中的语言建模任务通常依赖参数量达数十亿、需海量数据和巨额算力训练的大型模型。本研究聚焦语言建模的一个关键子任务——上下文编码，提出利用图卷积网络（GNN）对词语间的上下文关系进行建模，并与长短期记忆网络（LSTM）结合，基于前序词语序列预测下一个词。实验在自建维基百科文本语料库上进行，仅使用少量计算资源，结果表明该方法在有限条件下仍能实现较优的词预测表现。
原文摘要 · Abstract (English)
Language Modeling is a prevalent task in Natural Language Processing. The currently existing most recent and most successful language models often tend to build a massive model with billions of parameters, feed in a tremendous amount of text data, and train with enormous computation resources which require millions of dollars. In this project, we aim to address an important sub-task in language modeling, i.e., context embedding. We propose an approach to exploit the Graph Convolution operation in GNNs to encode the context and use it in coalition with LSTMs to predict the next word given a local context of preceding words. We test this on the custom Wikipedia text corpus using a very limited amount of resources and show that this approach works fairly well to predict the next word.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。