动态提取模型语义成分，让相似度计算更准更省
DySem: Uncovering Dynamic Semantic Components of Large Language Models for Calculating Semantic Textual Similarity
- 通过多语言共识找语义相关组件，动态生成每对文本的专属语义维度
- 在多个大模型上超越现有方法，且所需维度更低
- 适合需要高效精准文本相似度的场景，如检索与问答
计算语义文本相似度是自然语言处理的基础任务。当前基于大语言模型的方法通常依赖固定维度的最后层隐藏状态来计算文本对之间的相似度。我们指出该范式存在两大局限：(i) 最后一层隐藏状态编码的是更通用的知识而非纯粹的语义信息，不利于语义相似度计算；(ii) 大语言模型的隐藏层维度普遍很大，导致语义表示中存在冗余和噪声。本文提出 DySem，一种无需训练的新框架，通过多语言一致性识别模型内部更相关的语义组件，并摒弃静态表示空间，转而构建文本依赖的联合语义集合，基于共享的动态维度子集进行相似度计算。在多种大语言模型上的大量实验表明，本方法持续优于近期基线，同时降低相似度计算所需的维度。代码已开源：https://github.com/szu-tera/DySem。
原文摘要 · Abstract (English)
Calculating semantic textual similarity is a foundational task in natural language processing. Current large language models (LLMs) based methods typically rely on extracting last-layer hidden states with fixed dimensions to compute similarity for every text pairs. We argue that this paradigm is suffer from two limitations: (i) The last hidden layer encodes more general knowledge rather than just semantic knowledge, making it suboptimal for semantic similarity computation; (ii) The hidden layer dimensions of LLMs are generally very large, which introduces some redundancy and noise for representing semantics. In this work, we propose DySem, a novel training-free framework that investigates more semantic-related internal components of LLMs via multilingual consensus, and shifts away from static representation spaces in favor of dynamic, sample-specific semantic dimensions by constructing text-dependent joint semantic set and computes similarity over this shared dimensional subset. Extensive experiments across various LLMs show that our method consistently outperforms recent baselines while maintaining lower dimensions for similarity calculation. The code is released at https://github.com/szu-tera/DySem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。