arXiv:2606.31845cs.CLcs.LG2026-06被引 2

让Transformer的前馈层具备可读逻辑,自动识别语法许可关系。

Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors

论文配图:Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
图 1 · 摘自论文原文
  • 用模糊逻辑显式建模前馈层,引入可解释的“非”操作和量化机制。
  • 在125M模型上达到GELU基线水平的困惑度,且单元具可读语法功能。
  • 适合关注模型可解释性、语言逻辑结构的研究者使用。

Transformer的前馈层(FFN)虽能捕捉注意力提取的差异,却无法说明其计算内容。本文提出一种参数中立的替换:每个隐藏单元显式执行[0,1]区间上的模糊集合运算,包括交集A*B与差集A*(1-B),后者为有界正否定(“A但非B”),是门控/双线性单元所不具备的。在N位奇偶校验任务中,该方法在浅层具有最高参数效率;在125M规模(OpenWebText)下,其困惑度与GELU基线相当,每单位均携带明确逻辑形式。两个问题共享同一原因:双操作逻辑仅在第0层有效,训练中逐渐退化;而唯一稳定的语法缺陷集中于许可关系与量词,而非词内算子。为此,引入一个小型序列量词模块:软存在量词与软比例量词,各具每单元独立学习的遗忘率,源自粘滞初始化。该设计在第一轮即恢复缺陷(第二轮差距减半),在LAMBADA上略有领先,并使前馈层具备可读性:结构可迁移至深层,衰减机制解除粘滞(中位半衰期约1.5个词元;零锁存单元);在语义层,单元无需词典学习即可读作语法许可检测器:每个单元对许可词(如比较词、被动动词、否定极性词)激活,并将记忆传递以预测被许可词(than, by, nor)。该可读性局限于某一划分(完全布尔型FFN会训练发散),但最终实现参数中立、语言模型质量的Transformer,拥有从构建之初就可解释的语法机制——不仅说明前馈层表征了什么,更揭示其如何进行语法许可。

原文摘要 · Abstract (English)

A transformer's feed-forward (FFN) sublayer materializes the distinctions attention gathers, yet gives no account of what it computes. In a parameter-neutral replacement, each hidden unit is an explicit fuzzy set operation on sigmoid-bounded [0,1] memberships: intersection A*B and set-difference A*(1-B), the latter a bounded positive negation ("A but not B") that gated/bilinear units lack -- a negation-capable FFN (NC-FFN). On N-bit parity they are the most parameter-efficient reasoning basis at shallow depth; at scale (125M, OpenWebText) NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form. Two limits share one cause: two-operand logic localizes to layer 0 and erodes under training, and the one robust grammatical deficit concentrates in licensing and quantifiers, beyond within-token operators. We resolve both with a small block of sequence quantifiers: a soft existential and a soft proportion, each with a per-unit learned forgetting rate from a sticky init. This recovers the deficit at epoch one (halving the wider epoch-two gap), modestly leads on LAMBADA, and makes the FFN legible: the structure now holds and migrates into depth; the decay un-learns its stickiness (median half-life ~1.5 tokens; zero latch units); and at the semantic layers the units read, without dictionary learning, as grammatical licensing detectors: each fires on a licensor (a comparative, a passive participle, a negative-polarity item) and carries its memory forward to predict the licensed word (than, by, nor). This legibility is localized and free only up to a partition (a fully Boolean FFN diverges in training), but the result is a parameter-neutral, language-model-quality transformer with a readable, interpretable-by-construction grammatical mechanism -- an account not just of what a feed-forward layer represents but how it licenses.

可解释性语法推理模糊逻辑Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。