提出双向最大相似度方法，文档对齐速度提升100倍且精度不降。
BiMax: Bidirectional MaxSim Score for Document-Level Alignment
- 用双向最大相似度计算跨语言文档相似性，替代耗时的最优传输算法。
- 在WMT16任务上达到与最优传输相当的准确率，速度提升约100倍。
- 适用于大规模网页文档对齐，适合需高效处理多语种数据的研究者。
文档对齐是层次化网络数据挖掘的关键步骤，用于在同一网页领域内对齐源语言与目标语言的文档。已有高精度的基于句子嵌入的方法，如TK-PERT和最优传输（OT）。然而，面对海量网络挖掘数据，精度与速度均需兼顾。本文提出一种跨语言双向最大相似度（BiMax）评分方法，用于计算文档级相似性，相比OT方法显著提升效率。在WMT16双语文档对齐任务中，BiMax达到与OT相当的准确率，同时实现约100倍的速度提升。此外，本文还系统分析了当前主流多语言句子嵌入模型的表现。所有对齐方法均已开源，工具名为EmbDA（https://github.com/EternalEdenn/EmbDA）。
原文摘要 · Abstract (English)
Document alignment is necessary for the hierarchical mining (Bañón et al., 2020; Morishita et al., 2022), which aligns documents across source and target languages within the same web domain. Several high precision sentence embedding-based methods have been developed, such as TK-PERT (Thompson and Koehn, 2020) and Optimal Transport (OT) (Clark et al., 2019; El-Kishky and Guzmán, 2020). However, given the massive scale of web mining data, both accuracy and speed must be considered. In this paper, we propose a cross-lingual Bidirectional Maxsim score (BiMax) for computing doc-to-doc similarity, to improve efficiency compared to the OT method. Consequently, on the WMT16 bilingual document alignment task, BiMax attains accuracy comparable to OT with an approximate 100-fold speed increase. Meanwhile, we also conduct a comprehensive analysis to investigate the performance of current state-of-the-art multilingual sentence embedding models. All the alignment methods in this paper are publicly available as a tool called EmbDA (https://github.com/EternalEdenn/EmbDA).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。