评测大模型处理字符串的能力，发现其表现远不如人类。
StringLLM: Understanding the String Processing Capability of Large Language Models
- 构建StringBench数据集，系统评估大模型的字符串处理能力
- 大模型在字符串任务上准确率显著低于人类水平
- 通过微调提升模型表现，为后续研究提供基础
字符串处理是现代计算的核心组成部分，主要涉及字符串的分析与操作。尽管大型语言模型（LLM）在多种自然语言处理任务中取得显著进展，其在字符串处理方面的能力仍缺乏深入研究。为此，本文提出StringLLM方法，用于构建基准测试数据集以评估大模型的字符串处理能力。利用该方法，我们构建了名为StringBench的一系列数据集，涵盖广泛字符串处理任务，可系统评估大模型在该领域的表现。实验表明，大模型在字符串处理准确性上明显落后于人类。为进一步探究其原因，我们进行了深入分析，并提出一种有效的微调方法，显著提升了大模型的字符串处理能力。本工作为未来理解大模型字符串处理能力奠定了基础。代码与数据已开源：https://github.com/wxl-lxw/StringLLM。
原文摘要 · Abstract (English)
String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant advancements of large language models (LLMs) in various natural language processing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present a comprehensive study of LLMs' string processing capability. In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs. We use StringLLM to build a series of datasets, referred to as StringBench. It encompasses a wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and subsequently propose an effective approach that significantly enhances LLMs' string processing capability via fine-tuning. This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available at https://github.com/wxl-lxw/StringLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。