针对法规文本检索难题，提出多阶段调优策略提升准确率。
MST-R: Multi-Stage Tuning for Retrieval Systems and Metric Evaluation
- 分三步调优：硬负样本微调编码器、稀疏稠密混合检索、仅微调前k个召回结果。
- 在RegNLP挑战赛中取得领先成绩，显著提升检索性能。
- 揭示回答模型评分陷阱，为未来评估提供关键启示。
法规文档富含细微术语与专业语义，冻结检索增强生成系统（FRAG）因使用预训练（或冻结）组件，在检索与答问性能上面临挑战。本文提出一种多阶段调优（MST-R）策略，通过三阶段优化提升目标领域检索效果：(a) 使用硬负样本挖掘微调向量存储中的编码器；(b) 采用稀疏与稠密检索结合的混合检索方法，通过倒数排名融合（RRF）整合结果；(c) 仅对前k个召回结果进行交叉注意力编码器微调。我们在COLING 2025 RegNLP研讨会发布的RIRAG数据集上进行基准测试，系统在该挑战赛排行榜中获得最高得分。此外，我们发现一种简单答问方式能超越所有基线及预训练Llama模型，显著优于RePASs指标表现。分析该异常现象后，提出了对未来研究的重要建议。
原文摘要 · Abstract (English)
Regulatory documents are rich in nuanced terminology and specialized semantics. FRAG systems: Frozen retrieval-augmented generators utilizing pre-trained (or, frozen) components face consequent challenges with both retriever and answering performance. We present a system that adapts the retriever performance to the target domain using a multi-stage tuning (MST) strategy. Our retrieval approach, called MST-R (a) first fine-tunes encoders used in vector stores using hard negative mining, (b) then uses a hybrid retriever, combining sparse and dense retrievers using reciprocal rank fusion, and then (c) adapts the cross-attention encoder by fine-tuning only the top-k retrieved results. We benchmark the system performance on the dataset released for the RIRAG challenge (as part of the RegNLP workshop at COLING 2025). We achieve significant performance gains obtaining a top rank on the RegNLP challenge leaderboard. We also show that a trivial answering approach games the RePASs metric outscoring all baselines and a pre-trained Llama model. Analyzing this anomaly, we present important takeaways for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。