通过虚拟宽度提升模型表达力，加速训练且不增加计算开销。
Virtual Width Networks
- 将表示宽度与主干宽度解耦，扩大嵌入空间但保持计算量基本不变。
- 8倍虚拟宽度使下一项预测优化速度提升2倍，下两项预测提升3倍。
- 虚拟宽度越大越高效，适合大规模语言模型的高效训练设计。
我们提出虚拟宽度网络（VWN），在不带来隐藏层大小二次增长成本的前提下，实现更宽表示的优势。VWN将表示宽度与主干宽度解耦，在扩展嵌入空间的同时，几乎保持主干计算量不变。大规模实验表明，8倍虚拟宽度可使下一词预测的优化速度提升超过2倍，下一两词预测提升3倍。随着训练进行，损失差距增大，收敛加速比持续上升，说明VWN不仅具有令牌效率，而且在规模增大时愈发有效。此外，我们发现虚拟宽度与损失降低之间存在近似对数线性关系，为探索虚拟宽度缩放作为大模型效率的新维度提供了初步实证基础。
原文摘要 · Abstract (English)
We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。