对比大模型与BERT在中文量词预测上的表现，发现大模型反而更差。
From BERT to LLMs: Comparing and Understanding Chinese Classifier Prediction in Language Models
- 用掩码策略测试模型对量词的内在判断能力
- 大模型即使微调后仍不如BERT，量词预测表现差
- 量词预测依赖后接名词信息，双向注意力机制有优势
量词是汉语的重要特征，其准确预测对诸多教育应用至关重要。然而，当前主流大语言模型（LLMs）是否具备正确的汉语量词知识，尚未在自然语言处理领域得到充分研究。为此，我们采用多种掩码策略，评估大模型的内在能力、句子成分的贡献以及注意力机制的工作方式。同时探索了微调对提升量词预测性能的作用。结果表明，大模型在量词预测上表现甚至不如BERT，且预测效果显著依赖于后续名词的信息，这解释了像BERT这类具有双向注意力机制的模型为何更具优势。
原文摘要 · Abstract (English)
Classifiers are an important and defining feature of the Chinese language, and their correct prediction is key to numerous educational applications. Yet, whether the most popular Large Language Models (LLMs) possess proper knowledge the Chinese classifiers is an issue that has largely remain unexplored in the Natural Language Processing (NLP) literature. To address such a question, we employ various masking strategies to evaluate the LLMs' intrinsic ability, the contribution of different sentence elements, and the working of the attention mechanisms during prediction. Besides, we explore fine-tuning for LLMs to enhance the classifier performance. Our findings reveal that LLMs perform worse than BERT, even with fine-tuning. The prediction, as expected, greatly benefits from the information about the following noun, which also explains the advantage of models with a bidirectional attention mechanism such as BERT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。