微小的分词方式差异竟能让模型表现差11%，影响评测可靠性。
Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs
- 将冒号后空格与答案字母合并分词，显著提升准确率
- 不同分词策略导致最高11%的准确率差异和排名变化
- 该方法提升模型置信度校准，适合严谨评测者参考
在使用大语言模型（LLMs）进行多选题问答（MCQA）评估时，通常在提示末尾添加字符串"Answer:"，以便通过下一个词的概率自动提取答案。然而，对冒号后空格的分词方式缺乏共识，常被视作无关紧要的选择。本文揭示，这种看似无关的分词差异可导致高达11%的准确率波动，并引发模型排名的重排，引发对以往研究评测可靠性的担忧。令人意外的是，我们发现一种特定策略——将冒号后的空格与答案字母一起分词——能持续带来统计上显著的性能提升。此外，该策略还改善了模型校准，增强了置信度估计的可靠性。研究强调了评估设计的重要性，呼吁建立标准化、透明的评测协议以确保结果的可靠性和可比性。
原文摘要 · Abstract (English)
When evaluating large language models (LLMs) with multiple-choice question answering (MCQA), it is common to end the prompt with the string "Answer:" to facilitate automated answer extraction via next-token probabilities. However, there is no consensus on how to tokenize the space following the colon, often overlooked as a trivial choice. In this paper, we uncover accuracy differences of up to 11% due to this (seemingly irrelevant) tokenization variation as well as reshuffled model rankings, raising concerns about the reliability of LLM comparisons in prior work. Surprisingly, we are able to recommend one specific strategy -- tokenizing the space together with the answer letter -- as we observe consistent and statistically significant performance improvements. Additionally, it improves model calibration, enhancing the reliability of the model's confidence estimates. Our findings underscore the importance of careful evaluation design and highlight the need for standardized, transparent evaluation protocols to ensure reliable and comparable results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。