arXiv:2603.13271cs.CYcs.CL2026-03

梳理近70年词向量技术演进,揭示大模型时代研究范式巨变。

Tracing the Evolution of Word Embedding Techniques in Natural Language Processing

  • 按时间脉络分析四类词向量方法及其发展关系。
  • 大模型出现后,上下文相关方法占比提升6.4倍,团队规模显著扩大。
  • 30种新方法涌现,54种旧方法被弃用,产业参与度明显上升。

本文追溯了自然语言处理领域词向量技术的发展历程,系统收集并分析了1954至2025年间149篇研究论文,提供全面的方法论综述与数据驱动的文献计量分析,揭示表示学习在过去七十年中的演变。研究涵盖四类主要嵌入范式:基于统计的表示方法(如one-hot、bag-of-words、TF-IDF)、静态词向量(Word2Vec、GloVe、FastText)、上下文词向量(ELMo、BERT、GPT)以及句子/文档嵌入,深入讨论各类方法的优势、局限及其学术传承关系。除方法论回顾外,以GPT-3发布为分界线,采用七项假设检验量化研究焦点、合作模式与机构参与的变化。结果表明,大模型时代后,上下文与句子级方法的主导地位显著增强(优势比达6.4倍),团队平均规模显著增加(p=0.018),共出现30种全新技术,而54种预大模型方法未再被引用。结合产业参与度上升的证据,本研究为大语言模型如何重塑领域知识范式提供了定量依据。

原文摘要 · Abstract (English)

This work traces the evolution of word-embedding techniques within the natural language processing (NLP) literature. We collect and analyze 149 research articles spanning the period from 1954 to 2025, providing both a comprehensive methodological review and a data-driven bibliometric analysis of how representation learning has developed over seven decades. Our study covers four major embedding paradigms, statistical representation-based methods (one-hot encoding, bag-of-words, TF-IDF), static word embeddings (Word2Vec, GloVe, FastText), contextual word embeddings (ELMo, BERT, GPT), and sentence/document embeddings, critically discussing the strengths, limitations, and intellectual lineage connecting each category. Beyond the methodological survey, we conduct a formal era comparison using GPT-3's release as a dividing line, applying seven hypothesis tests to quantify shifts in research focus, collaboration patterns, and institutional involvement. Our analysis reveals a dramatic post-GPT-3 paradigm shift: contextual and sentence-level methods now dominate at 6.4X the odds of the pre-GPT-3 era, mean team sizes have grown significantly (p = 0.018), and 30 entirely new techniques have emerged while 54 pre-GPT-3 methods received no further attention. These findings, combined with evidence of rising industry involvement, provide a quantitative account of how the field's epistemic priorities have been reshaped by the advent of large language models.

词向量NLP演化大模型影响文献计量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。