arXiv:2601.00624cs.LG2026-01被引 6

测试大模型在简单问题上是否废话太多,给出量化评测标准。

Do Chatbot LLMs Talk Too Much? The YapBench Benchmark

  • 设计轻量级基准测试YapBench,用最小必要回答作参照
  • 76个模型平均多输出10倍字符,技术类请求最严重
  • 适合关注模型简洁性、成本控制的研究者和开发者

大型语言模型如ChatGPT、Claude和Gemini日益充当通用协作者,但在面对简单请求时常给出冗长回应,包含多余解释、模糊表述或固定套话,增加认知负担并提高按字符计费的推理成本。已有研究指出基于偏好的后训练和模型评分会引发系统性长度偏好,即更长的回答被奖励,即使质量相当。本文提出YapBench,一个用于量化用户可见过生成现象的轻量级基准。每个测试项包含单轮提示、精心挑选的最小必要基准答案及类别标签。核心指标YapScore衡量超出基准答案的字符数,支持跨模型比较且不依赖特定分词器。通过YapIndex——各类别中位YapScore的均匀加权平均值,总结模型表现。YapBench涵盖300多个英文提示,覆盖三类理想简洁场景:(A) 输入极简或模糊,理想响应为简短澄清;(B) 封闭式事实问题,答案简短稳定;(C) 一行代码任务,仅需单一命令或代码片段。对76个助手型LLM的评估显示,中位超额长度相差达一个数量级,且存在类别特异性失败模式:模糊输入下出现‘填空’现象,技术请求中存在解释或格式化开销。项目已开源,并维护实时排行榜以追踪模型冗余行为变化。

原文摘要 · Abstract (English)

Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini increasingly act as general-purpose copilots, yet they often respond with unnecessary length on simple requests, adding redundant explanations, hedging, or boilerplate that increases cognitive load and inflates token-based inference cost. Prior work suggests that preference-based post-training and LLM-judged evaluations can induce systematic length bias, where longer answers are rewarded even at comparable quality. We introduce YapBench, a lightweight benchmark for quantifying user-visible over-generation on brevity-ideal prompts. Each item consists of a single-turn prompt, a curated minimal-sufficient baseline answer, and a category label. Our primary metric, YapScore, measures excess response length beyond the baseline in characters, enabling comparisons across models without relying on any specific tokenizer. We summarize model performance via the YapIndex, a uniformly weighted average of category-level median YapScores. YapBench contains over three hundred English prompts spanning three common brevity-ideal settings: (A) minimal or ambiguous inputs where the ideal behavior is a short clarification, (B) closed-form factual questions with short stable answers, and (C) one-line coding tasks where a single command or snippet suffices. Evaluating 76 assistant LLMs, we observe an order-of-magnitude spread in median excess length and distinct category-specific failure modes, including vacuum-filling on ambiguous inputs and explanation or formatting overhead on one-line technical requests. We release the benchmark and maintain a live leaderboard for tracking verbosity behavior over time.

大模型评测模型简洁性基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。