用知识图谱生成可调难度的多选题,省时省钱还靠谱。
KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration
- 基于知识图谱构建可复用的结构化信息,支持按需生成题目
- 在历史、生物、数学领域生成6个数据集,质量五项指标全达标
- 适合需要可控难度评估的AI模型测试与教育应用
随着大语言模型(LLMs)的发展,它们在检索增强生成(RAG)等应用中日益重要,但系统评估仍受限于专用评测数据集的构建成本。本文提出KNIGHT,一种基于大语言模型、以知识图谱驱动的多选题(MCQ)自动生成框架,能从外部来源生成题目。KNIGHT构建特定主题的知识图谱,作为实体与关系的紧凑结构化摘要,可重复用于生成不同难度的题目(包括多跳问题),无需反复输入原始文本。该知识图谱作为压缩后的可重用状态,使题目生成仅需低成本读取。我们在维基百科/维基数据上实现该框架,保持领域和本体无关性。案例研究生成了历史、生物、数学领域的6个MCQ数据集。通过流畅性、无歧义性(唯一正确答案)、主题相关性、选项唯一性和基于源文档的答案可回答性(作为幻觉代理)五个维度评估。结果表明,KNIGHT实现了高效的令牌与成本生成,各项质量指标表现优异,模型排名与MMLU基准一致,支持主题与难度可控的评估。
原文摘要 · Abstract (English)
With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured and parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This knowledge graph acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable graph representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。