用提示调优让编码器模型跨语言零样本泛化，效率更高
Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models
- 将任务转为固定模板，通过提示调优实现跨语言零样本推理
- 多语言编码器在多个基准上媲美大型多语言模型，但计算成本更低
- 适合资源匮乏语言的高效NLP应用，推动更普惠的模型部署
大型语言模型在零样本和少样本任务中表现优异，但编码器类模型如BERT和RoBERTa因架构限制，难以达到类似效果。然而编码器具有计算和内存开销低的优势。近期工作通过提示调优（Statement Tuning）将任务重构为有限模板，使编码器实现零样本泛化。本文将其拓展至多语言自然语言处理，探究编码器能否实现跨语言零样本泛化，并作为内存密集型大模型在低资源语言中的高效替代方案。结果表明，当前最先进的编码器模型在跨语言任务中表现出色，性能可与多语言大模型比肩，同时更具效率。我们还分析了多语言提示调优的数据集设计、效率提升及语言特异性泛化能力，为构建更包容、资源友好的NLP模型提供支持。代码与模型已公开。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel in zero-shot and few-shot tasks, but achieving similar performance with encoder-only models like BERT and RoBERTa has been challenging due to their architecture. However, encoders offer advantages such as lower computational and memory costs. Recent work adapts them for zero-shot generalization using Statement Tuning, which reformulates tasks into finite templates. We extend this approach to multilingual NLP, exploring whether encoders can achieve zero-shot cross-lingual generalization and serve as efficient alternatives to memory-intensive LLMs for low-resource languages. Our results show that state-of-the-art encoder models generalize well across languages, rivaling multilingual LLMs while being more efficient. We also analyze multilingual Statement Tuning dataset design, efficiency gains, and language-specific generalization, contributing to more inclusive and resource-efficient NLP models. We release our code and models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。