通过动态更新的后验知识,提升多跳检索的准确性。
Momentum Posterior Regularization for Multi-hop Dense Retrieval
- 用前一跳和当前跳的黄金知识生成查询聚焦摘要作为后验信息。
- 采用动量平均策略同步更新前后置检索器,使知识蒸馏更稳定。
- 在HotpotQA和StrategyQA上显著优于现有方法,适合多跳问答场景。
多跳问答(Multi-hop QA)需要按顺序进行知识检索,每一步基于前序步骤的信息补充缺失知识。为提升检索效率,本文旨在将拥有答案等后验信息的后置检索器的知识,迁移到推理阶段无后验信息的前置检索器中。然而,现有知识蒸馏方法在多跳场景下表现不佳,主要因两点:1)后验信息常被定义为最终答案,缺乏与查询的直接关联;2)前置与后置检索器间存在巨大知识鸿沟,导致蒸馏过程不稳定甚至性能下降。为此,本文提出MoPo(Momentum Posterior Regularization),包含两项创新:1)将每一步的后验信息定义为前一跳与当前跳黄金知识的查询聚焦摘要;2)采用动量移动平均法同步更新前后置检索器,实现更平滑有效的知识迁移。在HotpotQA和StrategyQA上的实验表明,MoPo在检索与下游问答任务中均优于现有基线。
原文摘要 · Abstract (English)
Multi-hop question answering (QA) often requires sequential retrieval (multi-hop retrieval), where each hop retrieves missing knowledge based on information from previous hops. To facilitate more effective retrieval, we aim to distill knowledge from a posterior retrieval, which has access to posterior information like an answer, into a prior retrieval used during inference when such information is unavailable. Unfortunately, current methods for knowledge distillation in one-time retrieval are ineffective for multi-hop QA due to two issues: 1) Posterior information is often defined as the response (i.e. the answer), which may not clearly connect to the query without intermediate retrieval; and 2) The large knowledge gap between prior and posterior retrievals makes existing distillation methods unstable, even resulting in performance loss. As such, we propose MoPo (Momentum Posterior Regularization) with two key innovations: 1) Posterior information of one hop is defined as a query-focus summary from the golden knowledge of the previous and current hops; 2) We develop an effective training strategy where the posterior retrieval is updated along with the prior retrieval via momentum moving average method, allowing smoother and effective distillation. Experiments on HotpotQA and StrategyQA demonstrate that MoPo outperforms existing baselines in both retrieval and downstream QA tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。