提出新注意力机制,更好捕捉句子间细微差异。
Comateformer: Combined Attention Transformer for Semantic Sentence Matching
- 设计组合式注意力,可加减或缩放向量构建表示
- 在10个数据集上均实现一致性能提升
- 适合需要精细语义匹配的场景
基于Transformer的模型在语义匹配任务中通过捕捉词组对之间的关联取得了显著进展。然而,仅考察句子间的普遍相似性不足以评估其相关性,还需关注二者间的细微差别。遗憾的是,传统Transformer中的注意力softmax机制往往忽略这些细微差异。为此,本文提出一种新型语义句子匹配模型——基于Transformer的组合注意力网络(Comateformer)。该模型设计了一种具有组合性质的新型Transformer准注意力机制,不同于仅调整输入标记权重的传统方法,本方法学习如何组合、相减或缩放特定向量以构建表示。此外,该方法在计算双重亲和度得分时结合了相似性与不相似性(负亲和性)的直觉,从而更有效地表征句子间的关系。为评估模型性能,我们在十个公开的真实世界数据集上进行了广泛实验及鲁棒性测试,结果表明该方法实现了持续改进。
原文摘要 · Abstract (English)
The Transformer-based model have made significant strides in semantic matching tasks by capturing connections between phrase pairs. However, to assess the relevance of sentence pairs, it is insufficient to just examine the general similarity between the sentences. It is crucial to also consider the tiny subtleties that differentiate them from each other. Regrettably, attention softmax operations in transformers tend to miss these subtle differences. To this end, in this work, we propose a novel semantic sentence matching model named Combined Attention Network based on Transformer model (Comateformer). In Comateformer model, we design a novel transformer-based quasi-attention mechanism with compositional properties. Unlike traditional attention mechanisms that merely adjust the weights of input tokens, our proposed method learns how to combine, subtract, or resize specific vectors when building a representation. Moreover, our proposed approach builds on the intuition of similarity and dissimilarity (negative affinity) when calculating dual affinity scores. This allows for a more meaningful representation of relationships between sentences. To evaluate the performance of our proposed model, we conducted extensive experiments on ten public real-world datasets and robustness testing. Experimental results show that our method achieves consistent improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。