测试大模型在抽象常识推理中的表现,发现其能力仍有明显短板。
Navigating Semantic Relations: Challenges for Language Models in Abstract Common-Sense Reasoning
- 用概念网构建提示,分指令和少样本两种方式评估模型
- 单选时准确率显著下降,五选一比全集选择效果更好
- 提示工程可提升性能,适合研究常识推理的学者参考
大型语言模型在生成类人文本和解决中等复杂度推理任务方面表现优异,但在需要深层认知能力的任务如常识理解与抽象推理方面仍待深入探索。本文基于概念网知识图谱系统评估大模型的抽象常识推理能力,提出两种提示方法:指令提示(根据定义预测合理语义关系)和少样本提示(利用示例引导识别关系)。以gpt-4o-mini模型实验发现,在指令提示下,多选排序表现稳定,但单选时准确率大幅下降;在少样本提示下,从五个关系中选择比从完整集合中选择准确率显著提升,但存在对某些关系的显著偏好。结果表明,即使在商用大模型中,抽象常识推理能力仍远未达到人类水平。然而,研究也揭示了基于选择性检索的精心提示工程具有提升性能的潜力。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable performance in generating human-like text and solving reasoning tasks of moderate complexity, such as question-answering and mathematical problem-solving. However, their capabilities in tasks requiring deeper cognitive skills, such as common-sense understanding and abstract reasoning, remain under-explored. In this paper, we systematically evaluate abstract common-sense reasoning in LLMs using the ConceptNet knowledge graph. We propose two prompting approaches: instruct prompting, where models predict plausible semantic relationships based on provided definitions, and few-shot prompting, where models identify relations using examples as guidance. Our experiments with the gpt-4o-mini model show that in instruct prompting, consistent performance is obtained when ranking multiple relations but with substantial decline when the model is restricted to predicting only one relation. In few-shot prompting, the model's accuracy improves significantly when selecting from five relations rather than the full set, although with notable bias toward certain relations. These results suggest significant gaps still, even in commercially used LLMs' abstract common-sense reasoning abilities, compared to human-level understanding. However, the findings also highlight the promise of careful prompt engineering, based on selective retrieval, for obtaining better performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。