大模型计数性能受语义内容影响，同一任务不同对象结果差超40%。
Semantic Content Determines Algorithmic Performance
- 设计原子化测试集WhatCounts，仅改变计数对象语义
- 前沿大模型在不同语义下准确率相差超40%
- 小量无关微调即引发性能突变，适合关注模型鲁棒性者
计数不应依赖被计数的内容；更一般地，任何算法的行为应与输入的语义内容无关。我们提出WhatCounts来孤立测试这一性质。不同于以往将语义敏感性与推理复杂度或提示变化混为一谈的工作，WhatCounts是原子性的：在无歧义、有分隔的列表中计数，无重复、干扰项或推理步骤，仅改变语义类型。前沿大模型在仅改变计数对象（如城市与化学品、名字与符号）时，准确率差异超过40%。受控消融实验排除了混淆因素。该差距源于语义本身，且在少量无关微调后会不可预测地转移。大模型不实现算法，而是近似算法，而这种近似依赖于输入参数。我们通过一个代理示例表明，这不仅影响计数，任何大模型函数都可能隐含对输入意义的依赖。
原文摘要 · Abstract (English)
Counting should not depend on what is being counted; more generally, any algorithm's behavior should be invariant to the semantic content of its arguments. We introduce WhatCounts to test this property in isolation. Unlike prior work that conflates semantic sensitivity with reasoning complexity or prompt variation, WhatCounts is atomic: count items in an unambiguous, delimited list with no duplicates, distractors, or reasoning steps for different semantic types. Frontier LLMs show over 40% accuracy variation depending solely on what is being counted - cities versus chemicals, names versus symbols. Controlled ablations rule out confounds. The gap is semantic, and it shifts unpredictably with small amounts of unrelated fine-tuning. LLMs do not implement algorithms; they approximate them, and the approximation is argument-dependent. As we show with an agentic example, this has implications beyond counting: any LLM function may carry hidden dependencies on the meaning of its inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。