分离表征与重建，让文本编码器可高效扩展。
Separating Representation from Reconstruction Enables Scalable Text Encoders
- 将表征学习与重构任务解耦，设计双模块架构
- 支持超50%掩码率，吞吐提升1.5至2倍
- 适合需要高效训练和高样本效率的场景
尽管解码器快速扩展，编码器自BERT以来变化甚微。通过冻结主干评估与探针测试发现，尽管困惑度改善，BERT的表征对冻结探针越来越难以利用，根源在于其扁平结构将表征学习与标记重构损失耦合。本文提出CrossBERT，一种两部分架构，将高质量表征学习与严格的标记重构解耦。该设计支持高达50%以上的掩码率，并通过互补掩码策略实现所有标记的梯度传播，分别使吞吐量提升1.5至2倍，样本效率提高2倍。整体上，CrossBERT在MTEB(eng, v2)和冻结GLUE基准上表现出持续可扩展性与更优性能。
原文摘要 · Abstract (English)
While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly $\textit{unexploitable}$ by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose $\textbf{CrossBERT}$, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50\%$) and gradient collection over all tokens via a $\textit{Complementary Masking Strategy}$, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。