提出Bactrainus框架,提升大模型在多跳问答中的证据筛选能力。
Bactrainus: Optimizing Large Language Models for Multi-hop Complex Question Answering Tasks
- 模块化设计分离选段、找证据和生成答案三个步骤
- 在HotpotQA上达到89.01的答对F1,比仅用原始上下文高17-21点
- 适合需要可解释性与证据控制的复杂问答系统研究者
多跳问答需从多文档中识别并整合分散证据,但大语言模型仍易受无关上下文干扰。本文在英文HotpotQA的干扰设置下研究这一证据瓶颈,提出Bactrainus——一种模块化选择器-阅读器框架,分离段落选择、支持句识别与答案生成。通过可选的问题分解与教师生成的推理链监督,检验额外推理结构的作用。评估涵盖基础模型筛选、受控上下文与提示消融、对Llama 3.1 8B Instruct与70B Instruct阅读器的参数高效适配,以及集成选择器-阅读器实验。使用完整候选上下文而非黄金支持事实,导致答案令牌重叠F1下降17-21点,表明规模本身无法消除上下文敏感性。最大差异与阅读器适配及句子级证据控制相关。最强配置取得89.01的答对F1与79.70的联合F1,而问题分解与推理监督变体带来较小且依赖配方的改进。结果支持为固定候选多跳问答构建可审计、显式监督的证据接口,并呼吁对剩余微小差异进行盲测、匹配、多种子评估。
原文摘要 · Abstract (English)
Multi-hop question answering requires a system to identify and integrate evidence distributed across documents, yet large language models remain vulnerable to irrelevant context. We investigate this evidence bottleneck in the English HotpotQA distractor setting and introduce Bactrainus, a modular selector-reader framework that separates paragraph selection, supporting-sentence identification, and answer generation. Optional question decomposition and teacher-generated rationale supervision make it possible to test where additional reasoning structure is useful. The evaluation combines foundation-model screening, controlled context and prompting ablations, parameter-efficient adaptation of Llama 3.1 8B Instruct and Llama 3.1 70B Instruct readers, and integrated selector-reader experiments. Supplying the full candidate context instead of gold supporting facts reduces answer token-overlap F1 by 17-21 points, showing that scale alone does not remove context sensitivity. The largest observed differences are associated with reader adaptation and sentence-level evidence control. The strongest reported configuration obtains 89.01 answer F1 and 79.70 joint F1, whereas decomposition and rationale-supervision variants yield smaller, recipe-dependent changes. These findings support auditable, explicitly supervised evidence interfaces for fixed-candidate multi-hop QA and motivate blind, matched, multi-seed evaluation of the remaining small differences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。