让每个模型层都记得词元身份，解决罕见词训练不足和语义混淆问题。
TIDE: Every Layer Knows the Token Beneath the Context
- 引入嵌入记忆模块，每层动态注入词元身份信息
- 在多个语言任务上提升性能，罕见词表现显著改善
- 适合关注词表示细节与小样本任务的研究者
我们重新审视现代大模型中一个被普遍接受却未被深入研究的设计：词元索引仅在输入嵌入层查表一次后即被丢弃。这一单次注入假设导致两个结构性缺陷：(i) 罕见词问题——词汇分布呈齐普夫定律时，罕见词嵌入因接收的梯度信号远少于常见词而长期欠训练；(ii) 上下文坍塌问题——参数受限模型将分布相似的词映射为不可区分的隐藏状态。为此，我们提出TIDE，通过在标准Transformer中加入嵌入记忆（EmbeddingMemory）：一个由K个独立的MemoryBlock组成的集合，将词元索引映射为上下文无关的语义向量，一次性计算并经深度条件化的softmax路由器注入每一层，包含可学习的空值缓存。理论与实证均证明TIDE能有效缓解单一词元身份注入带来的问题，并在多个语言建模与下游任务中提升性能。
原文摘要 · Abstract (English)
We revisit a universally accepted but under-examined design choice in every modern LLM: a token index is looked up once at the input embedding layer and then permanently discarded. This single-injection assumption induces two structural failures: (i) the Rare Token Problem, where a Zipf-type distribution of vocabulary causes rare-token embeddings are chronically under-trained due to receiving a fraction of the cumulative gradient signal compared to common tokens; and (ii) the Contextual Collapse Problem, where limited parameters models map distributionally similar tokens to indistinguishable hidden states. As an attempt to address both, we propose TIDE, which augments the standard transformer with EmbeddingMemory: an ensemble of K independent MemoryBlocks that map token indices to context-free semantic vectors, computed once and injected into every layer through a depth-conditioned softmax router with a learnable null bank. We theoretically and empirically establish the benefits of TIDE in addressing the issues associated with single-token identity injection as well as improve performance across multiple language modeling and downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。