用辅助模型评估隐藏状态，提升大模型事实检索能力
Enhancing Fact Retrieval in PLMs through Truthfulness
- 引入辅助模型分析主模型隐藏状态，判断输入真实性
- 在多个掩码语言模型上实现最高33%的事实检索提升
- 适合关注大模型知识挖掘与可信推理的研究者
预训练语言模型（PLMs）在预训练阶段通过预测下一个词或缺失词，编码了大量世界知识。这些模型被视为可自然语言查询的软知识库，因此如何量化并提升其可提取的事实成为研究热点。已有方法致力于增强从PLM中提取事实的能力，近期研究表明，可以利用PLM的隐藏状态来判断输入内容的真实性。然而，如何利用这一特性改进事实检索仍待探索。本文提出使用一个辅助模型，基于主模型的隐藏状态表示来评估输入的真伪。我们在多个掩码式PLM上验证该方法，结果表明其能将事实检索性能提升最高达33%。研究揭示了利用隐藏状态表示改善大模型事实知识检索的巨大潜力。
原文摘要 · Abstract (English)
Pre-trained Language Models (PLMs) encode various facts about the world at their pre-training phase as they are trained to predict the next or missing word in a sentence. There has a been an interest in quantifying and improving the amount of facts that can be extracted from PLMs, as they have been envisioned to act as soft knowledge bases, which can be queried in natural language. Different approaches exist to enhance fact retrieval from PLM. Recent work shows that the hidden states of PLMs can be leveraged to determine the truthfulness of the PLMs' inputs. Leveraging this finding to improve factual knowledge retrieval remains unexplored. In this work, we investigate the use of a helper model to improve fact retrieval. The helper model assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs. We evaluate this approach on several masked PLMs and show that it enhances fact retrieval by up to 33\%. Our findings highlight the potential of hidden states representations from PLMs in improving their factual knowledge retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。