根据层间几何变化动态分配FFN宽度，提升模型效率。
Geometry-Guided Layerwise FFN Width Allocation in Transformers
- 用几何度量评估每层的特征变换复杂度，指导宽度分配。
- 在7个预训练模型上，最优分配使验证损失降低，优于均匀或余弦调度。
- 适合追求高效架构设计的研究者与工程师。
前馈网络（FFN）占变压器模型大部分参数，但其隐藏层宽度通常固定不变。本文提出基于前向传播中各层行为测量来动态分配容量。将每个FFN视为传输一组标记表示，通过保持对应关系的平移、戈罗姆-沃瑟斯坦畸变以及一阶持久同调，在原始和尺度归一化度量下量化几何变化。构建分层近似代理模型，实现精确预算优化。在七个预训练语言模型中，原始欧氏距离工作量大致跟踪残差范数增长，而归一化后的工作量主要集中在前层。戈罗姆-沃瑟斯坦工作量比有限样本拓扑估计更一致地关联于扰动敏感性。在128M与256M参数量的成对训练中，若干归一化工作量调度方案相比均匀宽度和人工设计的余弦衰减，显著降低平均验证损失。在440M规模下，最佳几何基分配方案优于均匀分配远超余弦衰减，而反拓扑的原始控制反而劣于均匀分配。
原文摘要 · Abstract (English)
Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth. We ask whether this capacity can instead be allocated from a forward-pass measurement of layer behavior. We view each FFN as transporting a cloud of token representations and quantify the induced geometric change using correspondence-preserving shift, Gromov-Wasserstein distortion, and degree-one persistent homology under raw and scale-normalized metrics. A layerwise approximation surrogate yields an exact fixed-budget optimizer. Across seven pretrained language models, raw Euclidean work largely tracks residual-norm growth, whereas normalized work is predominantly front-loaded. Gromov-Wasserstein work is more consistently associated with perturbation-based layer sensitivity than the finite-sample topological estimate. In paired 128M and 256M training runs, several normalized-work schedules reduce mean validation loss relative to both uniform width and a hand-designed cosine taper. With the amplified paired differences at 440M, the best geometry-based allocations improve over uniform substantially larger than the cosine taper, while the anti-topological raw control is worse than uniform.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。