用三路乘积代替门控机制，提升小模型语言建模性能
TriPLU: Bypassing the Gate with Direct Trilinear Product FFNs in Tiny Language Models
- 提出三重坐标乘积结构，直接替代传统门控前馈层
- 在字符级数据上验证损失降至1.0637，优于多个对照组
- 适合低算力环境下微调小模型，但对优化策略敏感
我们研究小型解码器语言模型是否能从直接乘法学习特征投影的前馈层中获益。TriPLU是一种三重乘积线性单元，将常规门控前馈分支替换为仅含三路投影的三阶坐标乘积分支。在字符级TinyStories 1M字节前缀实验中，TriPLU的平均最佳验证损失为1.0637，低于相近的SwiGLU（1.1017）、四阶乘积对照组（1.0780）和二阶乘积对照组（1.1026）。在仅训练的Byte-BPE实验中，TriPLU在低学习率设置下也降低了TinyStories和WikiText-2原始数据的验证与保留集每字节比特数（BPB），PMI切片证据显示其在已见中高互信息相邻词对上表现更优。常学习率诊断表明，乘积分支归一化可缩小高学习率下的最佳检查点差距，但最终BPB仍会在热调度下下降。结论强调：在特定低算力场景下，直接乘积前馈层可改善固定预算小模型损失，但该分支优化敏感，未证明FLOP归一化效率、扩展规律或大模型泛化能力。
原文摘要 · Abstract (English)
We study whether tiny decoder-only language models benefit from feed-forward layers that directly multiply learned feature projections. TriPLU, a Trilinear Product Linear Unit, replaces the usual gated FFN branch with a product-only degree-3 branch that multiplies three projected streams coordinatewise. In a character-level TinyStories 1M-byte prefix study, TriPLU reaches a mean best validation loss of 1.0637, compared with 1.1017 for closely matched SwiGLU, 1.0780 for a degree-4 product control, and 1.1026 for a degree-2 control. In train-only Byte-BPE experiments, TriPLU also lowers validation and heldout bits per byte on TinyStories and WikiText-2 raw under low-learning-rate settings, with PMI-slice evidence suggesting gains on seen middle- and high-PMI adjacent-token pairs. Constant-learning-rate diagnostics show that product-branch normalization can reduce the high-learning-rate best-checkpoint gap, although final BPB still degrades under hot schedules. The resulting claim is deliberately narrow: direct product FFNs can improve fixed-budget small-model loss in specific low-compute regimes, but the branch is optimization-sensitive and does not establish FLOP-normalized efficiency, scaling behavior, or broad LLM performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。