通过剪枝减少层数，让SBERT模型更小更快，效果却不差。
Towards Building Efficient Sentence BERT Models using Layer Pruning
- 对BERT模型逐层剪枝，保留关键层提升效率
- 剪枝后模型性能接近完整版，且优于同尺寸新训练模型
- 适合资源有限语言的轻量级语义模型部署
本研究探讨了在构建高效句向量模型（SBERT）中应用层剪枝的有效性。目标是生成更小、计算复杂度更低的句子嵌入模型，同时保持强相似性表征能力。我们对比了剪枝前后的Muril和MahaBERT-v2模型，并与小型自训练模型（如MahaBERT-Small和MahaBERT-Smaller）进行评估。采用两阶段微调策略，依次基于自然语言推理（NLI）和语义文本相似性（STS）任务进行优化。结果表明，尽管层数减少，剪枝模型仍能保持与全模型相当的嵌入质量；且在相同规模下，剪枝模型始终优于从头训练的小模型。这证明层剪枝是一种有效构建轻量、高效嵌入模型的方法，可显著降低计算开销，提升对资源有限语言的支持能力。
原文摘要 · Abstract (English)
This study examines the effectiveness of layer pruning in creating efficient Sentence BERT (SBERT) models. Our goal is to create smaller sentence embedding models that reduce complexity while maintaining strong embedding similarity. We assess BERT models like Muril and MahaBERT-v2 before and after pruning, comparing them with smaller, scratch-trained models like MahaBERT-Small and MahaBERT-Smaller. Through a two-phase SBERT fine-tuning process involving Natural Language Inference (NLI) and Semantic Textual Similarity (STS), we evaluate the impact of layer reduction on embedding quality. Our findings show that pruned models, despite fewer layers, perform competitively with fully layered versions. Moreover, pruned models consistently outperform similarly sized, scratch-trained models, establishing layer pruning as an effective strategy for creating smaller, efficient embedding models. These results highlight layer pruning as a practical approach for reducing computational demand while preserving high-quality embeddings, making SBERT models more accessible for languages with limited technological resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。