检验大模型对短语语义的理解能力，发现其优于传统方法但不如微调模型。
Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models
- 用三个人工标注数据集测试大模型短语理解能力
- 大模型表现优于传统嵌入法，但不及微调模型
- 提示词策略效果不一，错误分析揭示理解局限
短语是人类表达语义的基本语言单位。本研究通过三个人工标注数据集，系统评估了基于API的大语言模型（LLM）在短语语义推理任务中的表现，采用自然语言指令进行引导，并探究少样本示范与思维链推理等常见提示技术的影响。结果显示，大模型在所有数据集上均显著优于传统嵌入方法；然而，其性能并未明显超越微调方法。高级提示策略的效果存在波动性。通过详细的错误分析，揭示了大模型在理解短语语义时面临的关键限制。代码与数据见 https://github.com/memray/llm_phrase_semantics。
原文摘要 · Abstract (English)
Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at https://github.com/memray/llm_phrase_semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。