用双路径线性变换提升Transformer效率，降低参数量同时保持性能。
Hybrid Dual-Path Linear Transformations for Efficient Transformer Architectures
- 将线性变换拆分为局部稀疏块与全局低秩瓶颈两路径。
- 在FineWeb-Edu上参数减少6.8%且验证损失更低。
- 适合追求高效模型设计或可解释性的研究者。
标准Transformer依赖密集的线性变换，将特征投影视为单一全秩操作，既低效又缺乏区分局部保留与全局整合的结构先验。为此，我们提出混合双路径线性（HDPL）算子，将仿射变换分解为两个拓扑不同的路径：用于高秩局部处理的稀疏块对角组件，以及用于全局上下文正则化的低秩变分自编码器（VAE）瓶颈。通过仅替换特定投影（查询、键、值、门控、上投影）为HDPL算子，而保留标准密集层用于聚合（输出、下投影），实现了效率与表征能力的更好平衡。在FineWeb-Edu数据集上的实验表明，该架构优于标准Llama风格基线，验证损失更低的同时参数量减少6.8%。此外，我们讨论了在Transformer主干中显式引入概率潜空间的重要意义，为推理时控制、超网络调控、持续适应、可解释性及跨模型/跨模态同步开辟新路径。代码已开源于https://github.com/VladimerKhasia/HDPL。
原文摘要 · Abstract (English)
Standard Transformer architectures rely heavily on dense linear transformations, treating feature projection as a monolithic, full-rank operation. We argue that this formulation is inefficient and lacks the structural inductive bias necessary for distinguishing between local feature preservation and global context integration. To address this, we introduce the Hybrid Dual-Path Linear (HDPL) operator, which decomposes the affine transformation into two topologically distinct pathways: a sparse block-diagonal component for high-rank local processing, and a low-rank Variational Autoencoder (VAE) bottleneck for global context regularization. By "surgically" replacing specific projections (Query, Key, Value, Gate, Up) with HDPL operators while retaining standard dense layers for aggregation (Output, Down), we achieve a superior balance of efficiency and representational power. Experiments on the FineWeb-Edu dataset demonstrate that the HDPL architecture outperforms a standard Llama-style baseline, reducing validation loss while simultaneously reducing parameter count by 6.8%. Beyond immediate performance gains, we discuss how the explicit materialization of a probabilistic latent space within the Transformer backbone serves as a vital architectural affordance, offering new pathways for inference-time or hypernetwork induced control, continual adaptation, interpretability, and cross-model or cross-modal synchronization. The code is available at https://github.com/VladimerKhasia/HDPL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。