提出新基准CUTE，测试大模型对词元拼写的理解能力
CUTE: Measuring LLMs' Understanding of Their Tokens
- 设计多种任务评估大模型对词元拼写的掌握程度
- 多数模型知道词元拼写但无法有效利用该信息
- 适合关注模型语言理解深度的研究者参考
大语言模型在各类任务中表现优异，通常将文本拆分为多字符词元并将其作为原子单位处理，而无法直接访问单个字符。这引发了一个问题：大模型能学到多少拼写知识？为此，我们提出了一个新基准CUTE，包含一系列旨在测试大模型拼写知识的任务。我们在CUTE上评估了多个主流大模型，发现大多数模型似乎掌握了其词元的拼写，却未能有效利用这些信息来操作文本，质疑了此类知识的泛化能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。