将大模型的领域知识提炼为简洁可验证的微观理论，提升回答可信度与准确率。
From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering
- 通过生成并压缩模型对主题的核心陈述，形成非冗余的微观理论
- 使模型在维基百科等语料上可落地答案数量提升8%，准确率提高8%绝对值
- 适合需要可解释、可验证知识的医疗等高可信场景
近期推理方法（如思维链、蕴含推理）虽能解释单个问题的回答逻辑，却难以揭示模型对主题的整体理解。本文提出将这种整体理解称为“微观理论”（microtheories），即一组系统性表述，共同支持对一系列问题的答案推导，从而增强信任并提升性能。方法是先用模型生成涵盖训练问题答案的语句构建知识库，再进行提炼，得到精炼、通用且无冗余的核心微观理论。实验表明，将这些理论加入通用语料（如维基百科）后，可补充语料中缺失的关键主题信息，使模型可落地的答案比例最多提升8%，答案准确率最高提升8%绝对值。在医疗领域的人类评估中，提炼后的微观理论包含更密集的主题关键事实。此外，我们引入$ p $-相关性概念量化微观理论对特定主题的覆盖程度。结果表明，微观理论是模型主题知识的有效浓缩，能有效增强现有语料，并提供性能提升与可解释的知识窗口。
原文摘要 · Abstract (English)
Recent reasoning methods (e.g., chain-of-thought, entailment reasoning) help users understand how language models (LMs) answer a single question, but they do little to reveal the LM's overall understanding, or "theory," about the question's topic, making it still hard to trust the model. Our goal is to materialize such theories - here called microtheories (a linguistic analog of logical microtheories) - as a set of sentences encapsulating an LM's core knowledge about a topic. These statements systematically work together to entail answers to a set of questions to both engender trust and improve performance. Our approach is to first populate a knowledge store with (model-generated) sentences that entail answers to training questions and then distill those down to a core microtheory that is concise, general, and non-redundant. We show that, when added to a general corpus (e.g., Wikipedia), microtheories can supply critical, topical information not necessarily present in the corpus, improving both a model's ability to ground its answers to verifiable knowledge (i.e., show how answers are systematically entailed by documents in the corpus, fully grounding up to +8% more answers), and the accuracy of those grounded answers (up to +8% absolute). We also show that, in a human evaluation in the medical domain, our distilled microtheories contain a significantly higher concentration of topically critical facts than the non-distilled knowledge store. Finally, we show we can quantify the coverage of a microtheory for a topic (characterized by a dataset) using a notion of $p$-relevance. Together, these suggest that microtheories are an efficient distillation of an LM's topic-relevant knowledge, that they can usefully augment existing corpora, and can provide both performance gains and an interpretable, verifiable window into the model's knowledge of a topic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。