测试大模型对字符级信息的理解能力，发现其在实际任务中表现不佳。
SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding
- 设计十个实用任务评估模型对子词的精准理解
- 九个主流大模型在字符计数等任务上表现差，平均准确率不足60%
- 适合关注模型基础能力、安全性和可解释性的研究者
大型语言模型虽大幅提升推理能力，但在字符级基础任务（如单词字母计数）上仍存在明显缺陷，根源在于分词机制。现有基准多聚焦简单字符操作，常被质疑缺乏现实意义。然而，文本地图导航、结构化表格解析等真实应用高度依赖精确的子词理解。为此，我们提出SubTokenTest，一个涵盖四个领域共十个实用任务的综合性评测基准，通过解耦复杂推理与分词错误，精准定位性能瓶颈。我们对九个先进LLM进行了全面评估，并研究了测试时扩展对子词推理的影响，以及字符信息在隐藏状态中的编码方式。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have significantly enhanced their reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-token understanding. In this regard, we introduce SubTokenTest, a comprehensive benchmark that assesses sub-token understanding through practical, utility-driven tasks. Our benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-token reasoning and explore how character-level information is encoded within the hidden states.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。