用哈希方法从预训练模型中无监督提取语法结构。
On Eliciting Syntax from Language Models via Hashing
- 将CKY算法升级为一阶哈希，统一编码词法与句法。
- 在多个数据集上达到可比肩监督方法的解析性能。
- 适合对低资源语法分析或模型内部机制研究者。
无监督分词（又称语法归纳）旨在从原始文本中推断句法结构。近期研究表明，二进制表示在词法和句法层面均具备出色的保信能力。本文探索利用该能力，仅依赖模型内隐语法结构，从原始文本中推导解析树的可行性。为此，我们将比特级CKY算法从零阶升级至一阶，以统一的二进制表示空间编码词法与句法；在对比哈希框架下，将训练方式由监督转为无监督，并引入一种新型损失函数，施加更强且均衡的对齐信号。实验表明，该模型在多个数据集上表现优异，证明其有效且高效，可在低成本下从预训练语言模型中获取高质量解析树。
原文摘要 · Abstract (English)
Unsupervised parsing, also known as grammar induction, aims to infer syntactic structure from raw text. Recently, binary representation has exhibited remarkable information-preserving capabilities at both lexicon and syntax levels. In this paper, we explore the possibility of leveraging this capability to deduce parsing trees from raw text, relying solely on the implicitly induced grammars within models. To achieve this, we upgrade the bit-level CKY from zero-order to first-order to encode the lexicon and syntax in a unified binary representation space, switch training from supervised to unsupervised under the contrastive hashing framework, and introduce a novel loss function to impose stronger yet balanced alignment signals. Our model shows competitive performance on various datasets, therefore, we claim that our method is effective and efficient enough to acquire high-quality parsing trees from pre-trained language models at a low cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。