通过语义偏移检测社区专属俚语和独特实体
Community-Specific Slang and Entity Detection via Semantic Shift in Fine-Tuned Language Models
- 用微调前后词向量差异识别社区特有词汇
- 底10%语义偏移词准确捕获社区俚语与专有名词
- 适合研究网络社群语言现象的研究者使用
我们提出一种无监督方法，通过识别词典中语义偏移最大的词汇来解析在线社区中的俚语、独特实体和民间用语。语义偏移定义为将预训练大模型（LLM）在特定社区文本语料上微调后，词语编码表示的变化。该值与基础模型与微调模型间词向量的余弦相似度成反比。我们在3个Reddit子版块（r/Technology、r/Gaming、r/WorldofWarcraft）的文本语料上微调DistilRoBERTa模型，对整个词典的余弦相似度分布建模，发现取底部10%的词能有效识别出具有社区特异性意义的词汇；而顶部10%的词则承载相对通用的语义。
原文摘要 · Abstract (English)
We propose an unsupervised method of resolving slang, unique entities, and folklore from online communities by isolating words in the lexicon that have the highest magnitude of semantic shift. Semantic shift is defined as the evolution of a word's encoded representation as a result of fine-tuning a pretrained Large Language Model (LLM) on a community-specific text corpus. This value is inversely proportional to the cosine similarity between the base model's encoded representation of a word, and a fine-tuned model's encoded representation. We fine-tune the DistilRoBERTa model on text corpora collected from 3 Reddit subreddits (r/Technology, r/Gaming, r/WorldofWarcraft), model a distribution of cosine similarity over the lexicon, and show that one can successfully resolve words that have unique significance to the community by pulling data in the bottom 10-percentile. In contrast, we show that data in the top 10-percentile consist of words that carry relatively universal semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。