Cobweb4L用多概念预测提升词掩码效果，训练更高效。
Incremental and Data-Efficient Concept Formation to Support Masked Word Prediction
- 基于信息论的类别效用和多概念机制生成预测
- 用少量数据达到甚至超过Word2Vec性能
- 适合小样本语言学习与快速增量训练场景
本文提出Cobweb4L，一种支持词掩码预测的高效语言模型学习方法。该方法基于增量式概念学习系统Cobweb，构建概率概念层次结构，每个概念存储与其标签相关的词频信息。系统通过属性值表示将词及其上下文编码为实例。Cobweb4L采用信息论变体的类别效用，并引入新性能机制，利用多个概念进行预测。实验表明，相比仅使用单一节点的传统机制，其性能显著提升。此外，Cobweb4L能快速学习，在相同任务中表现可媲美甚至优于Word2Vec。进一步实验显示，当训练数据较少时，Cobweb4L和Word2Vec的表现均优于BERT。未来工作将致力于使结论更具鲁棒性和包容性。
原文摘要 · Abstract (English)
This paper introduces Cobweb4L, a novel approach for efficient language model learning that supports masked word prediction. The approach builds on Cobweb, an incremental system that learns a hierarchy of probabilistic concepts. Each concept stores the frequencies of words that appear in instances tagged with that concept label. The system utilizes an attribute value representation to encode words and their surrounding context into instances. Cobweb4L uses the information theoretic variant of category utility and a new performance mechanism that leverages multiple concepts to generate predictions. We demonstrate that with these extensions it significantly outperforms prior Cobweb performance mechanisms that use only a single node to generate predictions. Further, we demonstrate that Cobweb4L learns rapidly and achieves performance comparable to and even superior to Word2Vec. Next, we show that Cobweb4L and Word2Vec outperform BERT in the same task with less training data. Finally, we discuss future work to make our conclusions more robust and inclusive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。