自动生成高难度多跳问答题,解决训练数据稀缺问题。
BMGQ: A Bottom-up Method for Generating Complex Multi-hop Reasoning Questions from Semi-structured Data
- 从半结构化数据出发,用自然语言推理构建证据簇。
- 反向构造问题,使单一线索无用但组合唯一指向答案。
- 两阶段质量评估保障题目难而可验证,适合模型训练。
构建真正考验模型检索与推理能力的训练级多跳问答(QA)数据集仍极具挑战。尽管已有少数评估数据集捕捉了‘难以搜索但易验证’的问题特征——需整合模糊、间接且跨领域的线索——但这类资源仍稀少,且多用于评估,不适合作为监督微调(SFT)或强化学习(RL)的训练数据。同时,手动构建非直接可检索的问题(即答案无法通过单次查询获得,需对松散关联的证据进行多跳推理)成本极高,难以扩展,成为训练强检索-推理智能体的关键瓶颈。为此,我们提出BMGQ,一种自下而上的自动化方法,从半结构化知识源生成高难度、可训练的多跳问题。BMGQ系统:(i)基于自然语言推理由关系类型驱动,实现多样化的逻辑标注证据簇扩展;(ii)采用反向问题生成,构造隐蔽线索,使孤立信息无用但组合后唯一确定目标实体;(iii)通过双阶段评估流程确保质量,结合多模型共识过滤与结构约束分解及基于证据的匹配。该过程可规模化,生成复杂、检索困难但可验证的问题,适用于SFT/RL训练及挑战性评估,大幅降低人工标注成本,同时保持强评估基准的难度特征。
原文摘要 · Abstract (English)
Building training-ready multi-hop question answering (QA) datasets that truly stress a model's retrieval and reasoning abilities remains highly challenging recently. While there have been a few recent evaluation datasets that capture the characteristics of hard-to-search but easy-to-verify problems -- requiring the integration of ambiguous, indirect, and cross-domain cues -- these data resources remain scarce and are mostly designed for evaluation, making them unsuitable for supervised fine-tuning (SFT) or reinforcement learning (RL). Meanwhile, manually curating non-trivially retrievable questions -- where answers cannot be found through a single direct query but instead require multi-hop reasoning over oblique and loosely connected evidence -- incurs prohibitive human costs and fails to scale, creating a critical data bottleneck for training high-capability retrieval-and-reasoning agents. To address this, we present BMGQ, a bottom-up automated method for generating high-difficulty, training-ready multi-hop questions from semi-structured knowledge sources. The BMGQ system (i) grows diverse, logically labeled evidence clusters through Natural Language Inference (NLI)-based relation typing and diversity-aware expansion; (ii) applies reverse question construction to compose oblique cues so that isolated signals are underinformative but their combination uniquely identifies the target entity; and (iii) enforces quality with a two-step evaluation pipeline that combines multi-model consensus filtering with structured constraint decomposition and evidence-based matching. The result is a scalable process that yields complex, retrieval-resistant yet verifiable questions suitable for SFT/RL training as well as challenging evaluation, substantially reducing human curation effort while preserving the difficulty profile of strong evaluation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。