测试视觉语言模型能否从描述钩针图案转为生成可执行步骤
CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?
- 用钩针领域专用语言作为中间表示，验证指令可执行性
- 模型在可执行正确性上表现远低于表面相似度，暴露长程推理缺陷
- 适合关注多模态模型实际操作能力的研究者参考
尽管多模态大模型能描述视觉内容，但其生成可执行流程的能力仍待探索。本文提出的CrochetBench通过钩针领域的细粒度流程推理，评估模型从描述到执行的转变：需识别针法、选择结构合理指令，并生成可编译的流程。采用钩针PARADE DSL作为中间表示，支持结构验证与功能执行评估。基准涵盖针法分类、指令定位，以及自然语言与图像到DSL的转换任务。所有任务中，随着评估标准从表面相似度转向可执行正确性，性能显著下降，暴露出长程符号推理和3D感知流程合成的局限。CrochetBench为评估多模态模型的流程能力提供了新视角，揭示了表面理解与真实创作场景中可执行精度之间的差距。代码已开源于https://github.com/Peiyu-Georgia-Li/crochetBench。
原文摘要 · Abstract (English)
While multimodal large language models can describe visual content, their ability to generate executable procedures remains underexplored. CrochetBench presented in this paper evaluates this shift from describing to doing through fine-grained procedural reasoning in crochet: models must recognize stitches, select structurally appropriate instructions, and generate compilable procedures. We adopt the CrochetPARADE DSL as our intermediate representation, enabling structural validation and functional evaluation via execution. The benchmark covers tasks including stitch classification, instruction grounding, and both natural language and image-to-DSL translation. Across all tasks, performance sharply decreases as the evaluation shifts from surface-level similarity to executable correctness, revealing limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. Our proposed CrochetBench offers a new lens for assessing procedural competence in multimodal models and highlights the gap between surface-level understanding and executable precision in real-world creative domains. Code is available at https://github.com/Peiyu-Georgia-Li/crochetBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。