用检索约束解码，发现大模型知识被低估了
Retrieval-Constrained Decoding Reveals Underestimated Parametric Knowledge in Language Models
- 用检索约束输出形式，避免因表达不同被误判错误
- 70B模型在标准解码下仅32.3%准确率，用新方法提至46.0%
- 小模型用新方法反而超过大模型，适合评估模型真实知识
语言模型蕴含大量事实知识，但常被判定为错误。我们假设这些答案其实正确，只是表达形式不同，被严格评估机制误判，导致对模型参数化知识的低估。为此提出检索约束解码（RCD），限制模型输出为唯一表面形式。构建包含19,137个常识问题的YAGO-QA数据集，评估135M至70B参数的开源语言模型。结果表明，标准解码严重低估模型知识：例如，Llama-3.1-70B在标准解码下仅32.3% F1，使用RCD后提升至46.0%；而Llama-3.1-8B在RCD下达33.0%，超越标准解码下的大模型表现。代码与数据已公开于https://github.com/Rajjaa/disambiguated-LLM。
原文摘要 · Abstract (English)
Language models (LMs) encode substantial factual knowledge, but often produce answers judged as incorrect. We hypothesize that many of these answers are actually correct, but are expressed in alternative surface forms that are dismissed due to an overly strict evaluation, leading to an underestimation of models' parametric knowledge. We propose Retrieval-Constrained Decoding (RCD), a decoding strategy that restricts model outputs to unique surface forms. We introduce YAGO-QA, a dataset of 19,137 general knowledge questions. Evaluating open-source LMs from 135M to 70B parameters, we show that standard decoding undervalues their knowledge. For instance, Llama-3.1-70B scores only 32.3% F1 with vanilla decoding but 46.0% with RCD. Similarly, Llama-3.1-8B reaches 33.0% with RCD, outperforming the larger model under vanilla decoding. We publicly share the code and dataset at https://github.com/Rajjaa/disambiguated-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。