对比Lion与AdamW优化器在跨编码器重排序中的表现，发现Lion更高效且效果更好。
Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT
- 用Lion和AdamW分别微调MiniLM、GTE、ModernBERT模型进行重排序。
- Lion使ModernBERT在TREC 2019上达NDCG@10 0.7225，MRR@10达0.5988。
- Lion提升GPU利用率2.67%~10.33%，适合资源敏感场景使用。
现代信息检索系统常采用两阶段流程：先快速初步召回，再通过计算密集的重排序阶段精炼结果。跨编码器因能深入分析查询-文档对，在重排序中表现出色。本文研究近期提出的Lion优化器在微调跨编码器重排序器时的影响。我们在MS MARCO段落排序数据集上，使用Lion和AdamW分别微调三个Transformer模型：MiniLM、GTE和ModernBERT（支持最长8192词元的上下文）。评估使用TREC 2019深度学习赛道和MS MARCO开发集（MRR@10）。实验在Modal云平台运行，结果显示，ModernBERT搭配Lion在TREC DL 2019上取得最佳性能，NDCG@10为0.7225，MAP为0.5121；而MiniLM搭配Lion在MS MARCO开发集上与ModernBERT并列，MRR@10均为0.5988。此外，Lion显著提升GPU效率，各模型利用率提升2.67%至10.33%。我们通过标准信息检索指标分析性能趋势，并讨论优化器对不同架构训练动态的影响。
原文摘要 · Abstract (English)
Modern information retrieval systems often employ a two-stage pipeline: an efficient initial retrieval stage followed by a computationally intensive reranking stage. Cross-encoders have shown strong effectiveness for reranking due to their deep analysis of query-document pairs. This paper studies the impact of the Lion optimizer, a recent alternative to AdamW, during fine-tuning of cross-encoder rerankers. We fine-tune three transformer models-MiniLM, GTE, and ModernBERT-on the MS MARCO passage ranking dataset using both optimizers. GTE and ModernBERT support extended context lengths (up to 8192 tokens). We evaluate effectiveness using TREC 2019 Deep Learning Track and MS MARCO dev set (MRR@10). Experiments, run on the Modal cloud platform, reveal that ModernBERT with Lion achieves the best NDCG@10 (0.7225) and MAP (0.5121) on TREC DL 2019, while MiniLM with Lion ties ModernBERT for MRR@10 (0.5988) on MS MARCO dev. Lion also provides superior GPU efficiency, improving utilization by 2.67% to 10.33% across models. We analyze performance trends using standard IR metrics and discuss the optimizer's impact on training dynamics across architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。