通过可微词汇剪枝，让模型生成更聚焦主题的文本。
Idea-Gated Transformers: Enforcing Semantic Coherence via Differentiable Vocabulary Pruning
- 分离语义规划与语法生成，用概念向量动态控制词汇选择。
- 在WikiText-103上保持相近困惑度，但主题保留率显著提升。
- 无需增加参数，就能有效防止生成偏离初始主题。
基于自回归语言模型（LLM）的下一词预测（NTP）常出现主题漂移问题，因过度依赖局部关联而缺乏全局规划。尽管扩大模型规模可缓解此问题，但NTP目标的根本局限依然存在。本文提出 Idea-Gated Transformer 架构，将语义规划与语法生成分离。引入辅助的 Idea Head 来预测未来上下文窗口的词袋分布，生成一个潜在的“概念向量”，用于实时调控主词汇表。设计了一种可微分的门控机制，动态抑制语义无关词汇，实现实时搜索空间剪枝。在 WikiText-103 上的实验表明，该模型与标准 GPT-2 基线相比达到相近的验证困惑度，但在领域保留能力上显著更优。定性与定量分析显示，门控机制能有效锁定生成于特定语义簇（如金融、科学），抵抗联想漂移，为实现更可控的语言建模提供一种参数高效路径。
原文摘要 · Abstract (English)
Autoregressive Language Models (LLMs) trained on Next-Token Prediction (NTP) often suffer from Topic Drift where the generation wanders away from the initial prompt due to a reliance on local associations rather than global planning. While scaling model size mitigates this, the fundamental myopia of the NTP objective remains. In this work, we introduce the Idea-Gated Transformer, a novel architecture that separates semantic planning from syntactic generation. We introduce an auxiliary Idea Head trained to predict the bag-of-words distribution for a future context window, creating a latent ``Concept Vector'' that actively gates the main vocabulary during generation. We propose a differentiable gating mechanism that suppresses semantically irrelevant tokens, effectively pruning the search space in real-time. Experiments on WikiText-103 demonstrate that while the Idea-Gated model achieves comparable validation perplexity to a standard GPT-2 baseline, it exhibits significantly superior Domain Retention. Qualitative and quantitative analysis reveals that the gating mechanism successfully locks generation into specific semantic clusters (e.g., Finance, Science) and resists associative drift, offering a parameter-efficient path toward more controllable language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。