线性输出层限制知识图谱嵌入模型表达能力，非线性可突破瓶颈。
On the Theoretical Limitations of Embedding-based Link Prediction
- 理论证明线性输出层存在秩瓶颈，影响模型表达能力。
- 非线性输出层在低参数开销下提升大型密集图的排名与拟合性能。
- 适用于需要扩展到大规模知识图谱的研究者与开发者。
神经网络常将低维嵌入映射到高维输出空间，通常使用线性输出层，这会引发“秩瓶颈”，限制模型的表征能力。此类瓶颈在知识图谱嵌入（KGE）中普遍存在，因实体输出空间规模可能远超嵌入维度。本文研究秩瓶颈对拟合训练数据的能力限制。不同于以往关注特定KGE所需嵌入维度的充分条件，我们推导出所有带线性输出层的KGE的必要条件，其随图大小与连通性增长。此外，我们引入基于混合的非线性输出层，在不显著增加参数量的前提下打破瓶颈。实验表明，该设计在大型密集数据集上显著提升排序性能与概率拟合效果，验证了理论预测。本工作揭示了线性输出层对KGE的局限性，并推动非线性替代方案用于大规模、高密度图的建模。
原文摘要 · Abstract (English)
Neural networks often map low-dimensional embeddings to high-dimensional output spaces. Usually, the output layer is linear, which can create a "rank bottleneck" that limits the functions a model can represent. Such bottlenecks are ubiquitous in link prediction models, such as knowledge graph embeddings (KGEs), as the output space of entities can be orders of magnitude larger than the embedding dimension. We investigate how rank bottlenecks limit model expressivity for fitting the training data. While previous work focused on sufficient bounds on the embedding dimension required for specific KGEs, we show necessary bounds for all KGEs with a linear output layer, which grow with graph size and connectivity. We also consider a non-linear output layer using mixtures to break the bottleneck without significant parameter overhead. Empirically, we show that models using this non-linear layer improve in ranking performance and probabilistic fit for large and dense datasets at a low parameter cost, as predicted by our theory. Our work reveals how linear output layers limit KGEs and motivates non-linear alternatives for scaling to large and dense graphs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。