WebFAQ 2.0扩展至108语言,含1.25M难负样本,提升多语密集检索效果。
WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval
- 通过网页直接爬取构建跨语言问答对,覆盖19800万条数据
- 新增1.25百万查询的难负样本,每查询含200个交叉编码分数
- 支持对比学习与知识蒸馏两种检索模型微调方法
我们推出WebFAQ 2.0,一个包含19800万条基于FAQ的自然问题-答案对的新版本数据集,覆盖108种语言。相比前版,其多语言覆盖范围和双语对齐的问答对数量显著增加,超过1430万,成为目前最大的基于FAQ资源。不同于原版,WebFAQ 2.0采用新型数据采集策略,直接从网页抓取并提取相关内容,通过页面标题与描述带来更丰富的上下文,提升了多样性。为响应社区反馈,我们还发布了用于训练密集检索器的难负样本数据集,涵盖20种语言的125万查询,这些负样本通过两阶段检索流程挖掘,并为每查询提供200个负样本的交叉编码分数。我们进一步展示了该资源如何支持两种主要的密集检索器微调策略:带MultipleNegativesRanking损失的对比学习,以及带MarginMSE损失的知识蒸馏。WebFAQ 2.0并非静态资源,而是长期计划的一部分。自2025年底起,结构化FAQ将通过Open Web Index定期发布,实现持续扩展与优化。我们已公开数据集及训练脚本,以促进多语言与跨语言信息检索研究。所有资源均可在GitHub和HuggingFace上免费获取。
原文摘要 · Abstract (English)
We introduce WebFAQ 2.0, a new version of the WebFAQ dataset, containing 198 million FAQ-based natural question-answer pairs across 108 languages. Compared to the previous version, it significantly expands multilingual coverage and the number of bilingual aligned QA pairs to over 14.3M, making it the largest FAQ-based resource. Unlike the original release, WebFAQ 2.0 uses a novel data collection strategy that directly crawls and extracts relevant web content, resulting in a substantially more diverse and multilingual dataset with richer context through page titles and descriptions. In response to community feedback, we also release a hard negatives dataset for training dense retrievers, with 1.25M queries across 20 languages. These hard negatives were mined using a two-stage retrieval pipeline and include cross-encoder scores for 200 negatives per query. We further show how this resource enables two primary fine-tuning strategies for dense retrievers: Contrastive Learning with MultipleNegativesRanking loss, and Knowledge Distillation with MarginMSE loss. WebFAQ 2.0 is not a static resource but part of a long-term effort. Since late 2025, structured FAQs are being regularly released through the Open Web Index, enabling continuous expansion and refinement. We publish the datasets and training scripts to facilitate further research in multilingual and cross-lingual IR. The dataset itself and all related resources are publicly available on GitHub and HuggingFace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。