构建带状态标注的日本食谱数据集，评测大模型对食材变化的理解能力。
A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task
- 构建结构化日文食谱数据集，精准标注食材状态变化。
- 设计三类新任务，验证大模型追踪食材状态的能力。
- 公开数据集，适合评估语言模型对现实世界动态的理解。
大型语言模型（LLMs）虽在海量流程文本上训练，却无法直接感知现实现象。以食谱为例，中间阶段的食材状态常被省略，导致模型难以准确追踪和理解。本文将状态探测（state probing）方法引入烹饪领域，提出新任务与数据集，评估LLMs识别烹饪过程中食材中间状态的能力。我们基于结构清晰、控制严格的日文食谱，构建了一个包含精确状态标注的新数据集。利用该数据集，设计了三项新任务，用于检验模型是否能追踪食材状态转换并识别中间步骤存在的食材。对Llama3.1-70B和Qwen2.5-72B等主流模型的实验表明，学习食材状态知识显著提升其对烹饪过程的理解，性能达到商用模型水平。数据集已公开于HuggingFace：https://huggingface.co/datasets/mashi6n/nhkrecipe-100-anno-1。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are trained on a vast amount of procedural texts, but they do not directly observe real-world phenomena. In the context of cooking recipes, this poses a challenge, as intermediate states of ingredients are often omitted, making it difficult for models to track ingredient states and understand recipes accurately. In this paper, we apply state probing, a method for evaluating a language model's understanding of the world, to the domain of cooking. We propose a new task and dataset for evaluating how well LLMs can recognize intermediate ingredient states during cooking procedures. We first construct a new Japanese recipe dataset with clear and accurate annotations of ingredient state changes, collected from well-structured and controlled recipe texts. Using this dataset, we design three novel tasks to evaluate whether LLMs can track ingredient state transitions and identify ingredients present at intermediate steps. Our experiments with widely used LLMs, such as Llama3.1-70B and Qwen2.5-72B, show that learning ingredient state knowledge improves their understanding of cooking processes, achieving performance comparable to commercial LLMs. The dataset are publicly available at: https://huggingface.co/datasets/mashi6n/nhkrecipe-100-anno-1
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。