用逻辑形式的图结构提升语言模型数据效率
Exploring Graph Representations of Logical Forms for Language Modeling
- 用图结构表示逻辑形式，构建新型语言模型
- 相同数据下性能远超BERT等文本模型
- 适合追求高效学习与知识内嵌的应用
我们主张基于逻辑形式的语言模型（LFLMs），认为其比传统文本模型更高效。为此，提出图结构形式化逻辑分布语义（GFoLDS）原型，作为LFLM的验证。实验表明，该模型能利用内置基础语言知识，快速学习复杂模式。在下游任务中，GFoLDS显著优于同数据预训练的文本型Transformer模型（如BERT），证明其可大幅降低数据需求。此外，性能随参数量和预训练数据增加而提升，显示其在真实场景中的可行性。
原文摘要 · Abstract (English)
We make the case for language models over logical forms (LFLMs), arguing that such models are more data-efficient than their textual counterparts. To that end, we introduce the Graph-based Formal-Logical Distributional Semantics (GFoLDS) prototype, a pretrained LM over graph representations of logical forms, as a proof-of-concept of LFLMs. Using GFoLDS, we present strong experimental evidence that LFLMs can leverage the built-in, basic linguistic knowledge inherent in such models to immediately begin learning more complex patterns. On downstream tasks, we show that GFoLDS vastly outperforms textual, transformer LMs (BERT) pretrained on the same data, indicating that LFLMs can learn with substantially less data than models over plain text. Furthermore, we show that the performance of this model is likely to scale with additional parameters and pretraining data, suggesting the viability of LFLMs in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。