Avey-B通过重构实现高效长文本编码，性能超越主流Transformer模型。
Avey-B
- 重构Avey为无注意力编码器，解耦静态与动态参数提升效率
- 在标准分类与检索任务中优于4种主流Transformer模型，长文本扩展性更优
- 适合资源受限场景下的工业级NLP应用，尤其关注长序列处理
紧凑的预训练双向编码器在计算和内存受限的工业自然语言处理中仍占主导地位。其有效性源于自注意力机制在序列级并行下实现高质量双向上下文建模，如BERT类架构所展示。近期提出的Avey是一种自回归、无注意力的替代方案，天然支持仅编码器形式。本文将Avey重新设计为仅编码器范式，并引入多项创新：解耦的静态与动态参数化、面向稳定性的归一化方法以及神经压缩技术。实验表明，该重构架构在标准词元分类与信息检索基准上显著优于四种广泛使用的基于Transformer的编码器，在长上下文场景下具有更强的可扩展性。
原文摘要 · Abstract (English)
Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。