arXiv:2506.00968cs.AI2025-06被引 2

用多头注意力融合局部与全局语义,提升词义消歧准确率。

PolyBERT: Fine-Tuned Poly Encoder BERT-Based Model for Word Sense Disambiguation

  • 采用多头注意力的多编码器融合局部和全局语义
  • 在F1分数上比基线模型高2%,减少37.6%的显存消耗
  • 适合需要高效精准词义消歧的研究者或工业应用

主流词义消歧(WSD)方法利用BERT从上下文和词义定义中提取语义,取得了显著性能。然而存在两大局限:一是未能平衡局部(标记级)与全局(序列级)语义表示,导致语义表征不足;二是训练时包含所有可能词义,造成不必要的计算开销。为此,本文提出基于多编码器的BERT模型PolyBERT,结合批对比学习(BCL)。首先,多编码器通过多头注意力融合局部与全局语义,增强表征能力;其次,引入BCL,将同一批中其他目标词的正确词义作为负样本,避免冗余输入,降低计算成本。实验表明,PolyBERT相比Huang的GlossBERT和Blevins的BEM在F1分数上提升2%;使用BCL后,相比无BCL版本节省37.6%的GPU小时。

原文摘要 · Abstract (English)

Mainstream Word Sense Disambiguation (WSD) approaches have employed BERT to extract semantics from both context and definitions of senses to determine the most suitable sense of a target word, achieving notable performance. However, there are two limitations in these approaches. First, previous studies failed to balance the representation of token-level (local) and sequence-level (global) semantics during feature extraction, leading to insufficient semantic representation and a performance bottleneck. Second, these approaches incorporated all possible senses of each target word during the training phase, leading to unnecessary computational costs. To overcome these limitations, this paper introduces a poly-encoder BERT-based model with batch contrastive learning for WSD, named PolyBERT. Compared with previous WSD methods, PolyBERT has two improvements: (1) A poly-encoder with a multi-head attention mechanism is utilized to fuse token-level (local) and sequence-level (global) semantics, rather than focusing on just one. This approach enriches semantic representation by balancing local and global semantics. (2) To avoid redundant training inputs, Batch Contrastive Learning (BCL) is introduced. BCL utilizes the correct senses of other target words in the same batch as negative samples for the current target word, which reduces training inputs and computational cost. The experimental results demonstrate that PolyBERT outperforms baseline WSD methods such as Huang's GlossBERT and Blevins's BEM by 2\% in F1-score. In addition, PolyBERT with BCL reduces GPU hours by 37.6\% compared with PolyBERT without BCL.

词义消歧BERT多编码器对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。