为波兰语设计可处理8192词元长文本的高效编码模型。
Long-Context Encoder Models for Polish Language Understanding
- 两阶段训练：先调整位置编码，再全参数持续预训练。
- 在25项任务上平均表现最佳，长文档任务显著优于现有模型。
- 适合需要理解长文本的波兰语应用，如金融分析、法律文本处理。
尽管解码器仅架构的大语言模型近期主导自然语言处理领域，但编码器仅架构仍是判别任务中成本更低、参数更高效的主流方案。然而，经典编码器如BERT受限于短上下文窗口，难以处理长文档。本文针对波兰语提出一种高质量模型，可处理最长8192词元的序列。该模型通过两阶段训练流程实现：先进行位置编码适配，再进行全参数连续预训练。此外，我们还提出了通过知识蒸馏训练的压缩模型变体。模型在25个任务上进行了评估，包括KLEJ基准、新提出的金融任务套件FinBench，以及其他需长文档理解的分类与回归任务。结果表明，该模型在波兰语及多语言模型中平均表现最优，在长上下文任务中显著超越竞争方案，同时在短文本任务上保持相当性能。
原文摘要 · Abstract (English)
While decoder-only Large Language Models (LLMs) have recently dominated the NLP landscape, encoder-only architectures remain a cost-effective and parameter-efficient standard for discriminative tasks. However, classic encoders like BERT are limited by a short context window, which is insufficient for processing long documents. In this paper, we address this limitation for the Polish by introducing a high-quality Polish model capable of processing sequences of up to 8192 tokens. The model was developed by employing a two-stage training procedure that involves positional embedding adaptation and full parameter continuous pre-training. Furthermore, we propose compressed model variants trained via knowledge distillation. The models were evaluated on 25 tasks, including the KLEJ benchmark, a newly introduced financial task suite (FinBench), and other classification and regression tasks, specifically those requiring long-document understanding. The results demonstrate that our model achieves the best average performance among Polish and multilingual models, significantly outperforming competitive solutions in long-context tasks while maintaining comparable quality on short texts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。