首次系统检验大模型对反向关系方向性的理解能力
Reversing Arrows in Large Language Models
- 构建包含5457条数据的基准测试集，评估大模型对反向关系的判断
- 发现五款开源模型在反向关系分类中存在系统性偏差
- 实体表示方式变化显著影响模型表现，提示需谨慎设计输入
大型语言模型（LLMs）在文本到知识图谱生成等任务上表现优异。然而，它们是否准确建模了反向关系的方向依赖语义仍不明确，即交换两个实体顺序会改变关系含义（如“母亲”与“孩子”）。据我们所知，本文是首个系统研究LLM中反向关系方向性的作品，采用包含5,457个实例、27种不同反向关系标签的基准测试。我们在多选提示框架下评估了五款开源LLMs，并通过替换原始实体为合成或掩码实体，进一步考察关系描述和实体表示的影响。结果表明，各模型在反向关系分类中存在系统性不对称；关系描述并非始终提升性能；模型表现对实体表示的变化敏感。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved strong performance on text-to-knowledge graph generation and related tasks. Nevertheless, it is still unclear whether they accurately model the direction-dependent semantics of inverse relations, in which reversing the order of the arguments alters the meaning of a relation (e.g., \textit{mother} versus \textit{child}). To the best of our knowledge, this work presents the first systematic study of inverse relation directionality in LLMs, using a benchmark consisting of 5,457 instances spanning 27 distinct inverse relation labels. We evaluate five open-source LLMs under a multiple-choice prompting framework and further examine the influence of relation descriptions and entity representations by substituting the original entities with synthetic and masked entities. Our findings reveal systematic asymmetries in inverse relation classification across LLMs, indicate that relation descriptions do not consistently improve performance, and show that model performance can be sensitive to variations in entity representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。