arXiv:2512.04125cs.LG2025-12中稿 · The Thirty-Ninth A…被引 4

用字符画测试大模型的空间理解能力,发现现有方法表现不佳。

ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text

  • 构建首个公开字符画评测基准ASCIIBench,含5315张标注图像。
  • 大模型生成的字符画在语义上难以区分,相似度接近随机水平。
  • 适合研究多模态表示与符号视觉模态的学者参考。

大型语言模型(LLMs)虽在长文本生成和推理方面展现出涌现能力,但在需要精确空间与位置推理的任务上仍存在困难。字符画作为一种以字符编码结构与形态的符号媒介,为检验该缺陷提供了独特途径。我们提出ASCIIBench,一个用于评估字符画生成与分类的新基准。该基准包含经筛选的5,315张带类别标签的字符画图像,据我们所知是首个公开可用的同类基准。同时,我们发布了针对字符画结构优化的微调版CLIP模型权重,支持对大模型生成的字符画进行评估。分析显示,基于CLIP嵌入的余弦相似度无法有效区分多数字符画类别,即使在低方差类别中也仅达到随机水平。相反,内部均值相似性高的类别则表现出明显可分性,揭示瓶颈在于表征而非生成方差。这些发现将字符画定位为多模态表征的强压力测试,并推动面向符号视觉模态的新型嵌入方法或评估指标的发展。所有资源均可在https://github.com/ASCIIBench/ASCIIBench获取。

原文摘要 · Abstract (English)

Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation. However, they continue to struggle with tasks requiring precise spatial and positional reasoning. ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation. We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images. ASCIIBench consists of a filtered dataset of 5,315 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind. Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art. Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes. In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance. These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities. All resources are available at https://github.com/ASCIIBench/ASCIIBench.

字符画多模态大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。