用波动向量表示文本，同时捕捉全局与局部语义，提升模型效率。
The Backpropagation of the Wave Network
- 用复数向量的模和相位分别表示全局语义与词间关系
- 相比BERT，显存占用和训练时间显著降低
- 适合追求高效推理的轻量级语言模型应用
本文深入分析了波网络（Wave Network）这一新型标记表示方法，其通过受波动启发的复数向量，同时捕捉输入文本的全局与局部语义。在复数向量表示中，每个标记由幅度分量（体现整体文本全局语义）和相位分量（编码个体标记与全局语义的关系）构成。基于先前研究证实波动操作（如干涉、调制）在前向传播中的有效性，本研究进一步探究了Token2Wave框架下的收敛行为、反向传播特性及嵌入独立性。计算复杂度分析表明，相较于BERT，Token2Wave可显著减少视频内存使用与训练时间。对[CLS]标记、总输入文本及分类器参数的梯度对比，凸显了其独特特征。该研究为基于波动的标记表示提供了新见解，展示了其在构建高效、计算友好的语言模型架构方面的潜力。
原文摘要 · Abstract (English)
This paper provides an in-depth analysis of Wave Network, a novel token representation method derived from the Wave Network, designed to capture both global and local semantics of input text through wave-inspired complex vectors. In complex vector token representation, each token is represented with a magnitude component, capturing the global semantics of the entire input text, and a phase component, encoding the relationships between individual tokens and the global semantics. Building on prior research that demonstrated the effectiveness of wave-like operations, such as interference and modulation, during forward propagation, this study investigates the convergence behavior, backpropagation characteristics, and embedding independence within the Token2Wave framework. A detailed computational complexity analysis shows that Token2Wave can significantly reduce video memory usage and training time compared to BERT. Gradient comparisons for the [CLS] token, total input text, and classifier parameters further highlight Token2Wave's unique characteristics. This research offers new insights into wave-based token representations, demonstrating their potential to enable efficient and computationally friendly language model architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。