测试大模型在长序列中重复简单规则的能力，发现性能随复杂度下降。
MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks
- 设计多语言长序列基准，可调控序列复杂度
- 模型在80项列表匹配任务中表现远低于20项时
- 非英语上下文示例能显著提升外推能力
语言模型已能处理包含数十万标记的长序列，但在需要重复应用简单规则的任务上表现不佳，即使序列远短于训练时所见。例如，当前最优大模型可处理最多20项的列表交集，但在80项时失败。本文提出MLissard，一个多语言基准，用于评估模型对不同长度文本的处理与生成能力，并支持控制序列复杂度。对开源与专有模型的评估显示，所有模型和语言在序列复杂度增加时性能持续下降。令人意外的是，使用非英语上下文示例能显著提升模型外推能力。数据集与代码已公开于https://github.com/unicamp-dl/Lissard。
原文摘要 · Abstract (English)
Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models' abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly. The datasets and code are available at https://github.com/unicamp-dl/Lissard
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。