arXiv:2506.23527cs.CL2025-06

分析大模型生成菜谱的抄袭、创意与荒诞,揭示其真实创作能力。

On Recipe Memorization and Creativity in Large Language Models: Is Your Model a Creative Cook, a Bad Cook, or Merely a Plagiator?

  • 用人工标注20道菜谱,识别哪些成分是直接复制,哪些是原创或胡编。
  • 发现模型常重复网络上的食材,说明严重依赖记忆而非真正创新。
  • 设计自动化流程,用模型当裁判,可大规模评估生成内容质量。

本研究探讨大型语言模型(LLMs)生成菜谱时的复现、创意与荒谬性。首先,我们对由Mixtral生成的20道预选菜谱进行细致的人工标注,提取每道菜的食材和步骤,判断其是否源自训练数据中的在线内容(即记忆),还是来自真实的创意合成或纯粹胡编。结果显示,Mixtral持续复用可在网络文档中找到的食材,表明对记忆内容有强烈依赖。为实现规模化分析,我们设计了“模型作为裁判”(LLM-as-judge)流水线,自动完成菜谱生成、荒谬性检测、成分与步骤解析及标注。例如,最佳成分提取与标注组合(Llama 3.1 + Gemma 2 9B)在成分匹配上达到最高78%准确率。该自动化框架实现了对生成菜谱中记忆、创意与荒谬性的大规模量化,提供了模型创造力的严谨证据。

原文摘要 · Abstract (English)

This work-in-progress investigates the memorization, creativity, and nonsense found in cooking recipes generated from Large Language Models (LLMs). Precisely, we aim (i) to analyze memorization, creativity, and non-sense in LLMs using a small, high-quality set of human judgments and (ii) to evaluate potential approaches to automate such a human annotation in order to scale our study to hundreds of recipes. To achieve (i), we conduct a detailed human annotation on 20 preselected recipes generated by LLM (Mixtral), extracting each recipe's ingredients and step-by-step actions to assess which elements are memorized--i.e., directly traceable to online sources possibly seen during training--and which arise from genuine creative synthesis or outright nonsense. We find that Mixtral consistently reuses ingredients that can be found in online documents, potentially seen during model training, suggesting strong reliance on memorized content. To achieve aim (ii) and scale our analysis beyond small sample sizes and single LLM validation, we design an ``LLM-as-judge'' pipeline that automates recipe generation, nonsense detection, parsing ingredients and recipe steps, and their annotation. For instance, comparing its output against human annotations, the best ingredient extractor and annotator is Llama 3.1+Gemma 2 9B, achieving up to 78% accuracy on ingredient matching. This automated framework enables large-scale quantification of memorization, creativity, and nonsense in generated recipes, providing rigorous evidence of the models' creative capacities.

大模型菜谱生成记忆机制自动化评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。