用可学习原型避免模型坍缩,提升Transformer编码器的压缩与表征能力。
Collapse-Free Prototype Readout Layer for Transformer Encoders
- 通过软概率匹配将输入分到少量全局原型向量,实现线性复杂度的压缩。
- 理论证明原型分离度随训练稳定增长,且损失分解为重建与多样性两部分。
- 适用场景广:文本/图像/科学数据均可,还能作为可微代码本或文档压缩器。
DDCL-Attention是一种基于原型的Transformer编码器读出层,替代传统的平均池化或类别标记,采用少量全局原型向量并通过软概率匹配分配令牌,以线性复杂度生成紧凑的令牌摘要。该方法有三大优势:一是通过训练损失的精确分解(重建项与多样性项)避免原型坍缩;二是利用Tikhonov奇异摄动理论和显式学习率约束,证明其与编码器联合训练在实际时间尺度下稳定;三是同一框架支持三种用途:最终读出层、可微代码本扩展VQ-VAE、层次化文档压缩器。四组数据集实验验证了理论预测:损失分解精确成立,满足稳定性条件时原型分离度按预期增长,代码本实现完全利用,优于标准硬向量量化。对轨道碎片分类的附加研究显示,该方法还可用于科学表格数据,超越传统NLP与视觉任务。
原文摘要 · Abstract (English)
DDCL-Attention is a prototype-based readout layer for transformer encoders that replaces simple pooling methods, such as mean pooling or class tokens, with a learned compression mechanism. It uses a small set of global prototype vectors and assigns tokens to them through soft probabilistic matching, producing compact token summaries at linear complexity in sequence length. The method offers three main advantages. First, it avoids prototype collapse through an exact decomposition of the training loss into a reconstruction term and a diversity term, ensuring that prototypes remain distinct. Second, its joint training with the encoder is shown to be stable under a practical timescale condition, using Tikhonov's singular perturbation theory and explicit learning-rate constraints. Third, the same framework supports three uses: a final readout layer, a differentiable codebook extending VQ-VAE, and a hierarchical document compressor. Experiments on four datasets confirm the theoretical predictions: the loss decomposition holds exactly, prototype separation grows as expected when the stability condition is met, and the codebook reaches full utilization, outperforming standard hard vector quantization. An additional study on orbital debris classification shows that the method also applies beyond standard NLP and vision tasks, including scientific tabular data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。