arXiv:2604.24940cs.CLcs.AI2026-04

用多向量嵌入提升词语表达力,让大模型更懂一词多义。

ADE: Adaptive Dictionary Embeddings -- Scaling Multi-Anchor Representations to Large Language Models

  • 将多锚点嵌入转化为单矩阵运算,大幅提升效率
  • 在DBpedia-14上超越DeBERTa-v3-base,参数减少98.7%
  • 适合追求高效语义表达的模型轻量化研究者

词嵌入是自然语言处理的基础,但传统方法为每个词分配单一向量,导致多义词表达受限。多锚点表示虽能改善语义表达,却因计算效率低且难以融入现代Transformer架构而仅限于小模型。本文提出自适应字典嵌入(ADE),首次将多锚点表示成功扩展至大语言模型。ADE包含三项核心创新:(1) 词汇投影(VP),将耗时的两阶段锚点查找简化为单次矩阵运算;(2) 分组位置编码(GPE),同一词的锚点共享位置信息,保持语义连贯性的同时支持锚点级差异;(3) 上下文感知锚点重加权,利用自注意力动态调整锚点贡献。我们将这些模块集成至分段感知变压器(SAT),实现在推理时根据上下文自适应重加权。在AG News和DBpedia-14文本分类任务上评估,与DeBERTa-v3-base相比,参数量减少98.7%,在DBpedia-14上达到98.06%准确率(优于原模型的97.80%),在AG News上达90.64%(接近原模型的94.50%),同时嵌入层压缩超40倍,证明多锚点表示是现代Transformer中高效、低参的可行替代方案。

原文摘要 · Abstract (English)

Word embeddings are fundamental to natural language processing, yet traditional approaches represent each word with a single vector, creating representational bottlenecks for polysemous words and limiting semantic expressiveness. While multi-anchor representations have shown promise by representing words as combinations of multiple vectors, they have been limited to small-scale models due to computational inefficiency and lack of integration with modern transformer architectures. We introduce Adaptive Dictionary Embeddings (ADE), a framework that successfully scales multi-anchor word representations to large language models. ADE makes three key contributions: (1) Vocabulary Projection (VP), which transforms the costly two-stage anchor lookup into a single efficient matrix operation; (2) Grouped Positional Encoding (GPE), a novel positional encoding scheme where anchors of the same word share positional information, preserving semantic coherence while enabling anchor-level variation; and (3) context-aware anchor reweighting, which leverages self-attention to dynamically compose anchor contributions based on sequence context. We integrate these components into the Segment-Aware Transformer (SAT), which provides context-aware reweighting of anchor contributions at inference time. We evaluate ADE on AG News and DBpedia-14 text classification benchmarks. With 98.7% fewer trainable parameters than DeBERTa-v3-base, ADE surpasses DeBERTa on DBpedia-14 (98.06% vs. 97.80%) and approaches it on AG News (90.64% vs. 94.50%), while compressing the embedding layer over 40x -- demonstrating that multi-anchor representations are a practical and parameter-efficient alternative to single-vector embeddings in modern transformer architectures.

词嵌入多锚点轻量化Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。