首个针对土耳其语信息检索的密集与延迟交互模型对比基准
TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrieval
- 用双阶段微调+PyLate转换,将英/多语言模型转为土耳其语检索器
- 小模型仅100万参数却保留大模型71%以上性能,延迟交互模型更优
- 支持低延迟部署,适合需快速响应的土耳其语应用开发
神经信息检索系统在高资源语言中表现优异,但在像土耳其语这样形态丰富、资源较少的语言上研究不足。当前密集双编码器主导土耳其语检索,但保留词级表示的延迟交互模型尚未被系统评估。我们提出TurkColBERT,首个全面比较密集编码器与延迟交互模型的土耳其语检索基准。通过两阶段适配流程,将英语和多语言编码器在土耳其语自然语言推理与语义相似性任务上微调,再用PyLate在MS MARCO-TR上训练转换为ColBERT风格检索器。在五个覆盖科学、金融和论辩领域的土耳其BEIR数据集上评估10个模型。结果表明:参数效率极高——100万参数的colbert-hash-nano-tr比6亿参数的turkish-e5-large小600倍,仍保持其平均mAP超过71%;延迟交互模型比密集模型小3–5倍,且在特定任务上最高提升13.8% mAP。生产可用性方面,MUVERA+Rerank索引算法比PLAID快3.33倍,相对mAP提升1.7%;ColmmBERT-base-TR在该方案下实现0.54毫秒查询延迟。所有检查点、配置与评测脚本均已开源。局限在于依赖中等规模数据集(≤5万文档)及翻译基准,可能无法完全反映真实场景,大规模MUVERA评估仍有待开展。
原文摘要 · Abstract (English)
Neural information retrieval systems excel in high-resource languages but remain underexplored for morphologically rich, lower-resource languages such as Turkish. Dense bi-encoders currently dominate Turkish IR, yet late-interaction models -- which retain token-level representations for fine-grained matching -- have not been systematically evaluated. We introduce TurkColBERT, the first comprehensive benchmark comparing dense encoders and late-interaction models for Turkish retrieval. Our two-stage adaptation pipeline fine-tunes English and multilingual encoders on Turkish NLI/STS tasks, then converts them into ColBERT-style retrievers using PyLate trained on MS MARCO-TR. We evaluate 10 models across five Turkish BEIR datasets covering scientific, financial, and argumentative domains. Results show strong parameter efficiency: the 1.0M-parameter colbert-hash-nano-tr is 600$\times$ smaller than the 600M turkish-e5-large dense encoder while preserving over 71\% of its average mAP. Late-interaction models that are 3--5$\times$ smaller than dense encoders significantly outperform them; ColmmBERT-base-TR yields up to +13.8\% mAP on domain-specific tasks. For production-readiness, we compare indexing algorithms: MUVERA+Rerank is 3.33$\times$ faster than PLAID and offers +1.7\% relative mAP gain. This enables low-latency retrieval, with ColmmBERT-base-TR achieving 0.54 ms query times under MUVERA. We release all checkpoints, configs, and evaluation scripts. Limitations include reliance on moderately sized datasets ($\leq$50K documents) and translated benchmarks, which may not fully reflect real-world Turkish retrieval conditions; larger-scale MUVERA evaluations remain necessary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。