分析5.7万条代码中的指令提示，揭示其使用规律与多样性。
Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code
- 从GitHub收集5.75万条可复现的代码指令提示，构建结构化分类体系。
- 发现提示使用呈现长尾分布，少数高频模式主导，多数模式罕见但多样。
- 开源数据集与可视化工具，适合研究提示工程、编程语言模型的开发者。
当代生成式大语言模型的行为直接受到提示（prompt）的影响，这些未结构化的自然语言文本描述了期望的输出和模型行为。本文主张提示本身是值得独立研究的语言对象。为此，我们从GitHub收集了57,500个唯一的提示样本，聚焦于交易型提示——可复现的自然语言指令，已集成于软件中。为支持提示的实证与量化研究，我们引入一个结构化本体，捕捉提示的形式与语义成分。基于此本体，将原始非结构化提示转化为丰富的结构化语言对象。对这些结构化数据的分析揭示：不同语言、领域、任务和模态间存在显著使用模式差异，典型地呈现类似齐普夫分布，部分模式明显占优，其余则分布于长尾。为验证本体标注的可靠性，我们在所有领域开展全面错误分析，评估标注质量。我们公开发布该数据集及浏览探索界面（https://github.com/OnlpLab/transactionalPromptsCollection）。
原文摘要 · Abstract (English)
The behavior of contemporary generative Large Language Models (LLMs) is directly shaped by prompts, unstructured texts that describe the desired output and model behavior. In this paper we argue that prompts are linguistic objects that merit investigation in their own right. To this end, we collect 57.5K unique samples of prompts from GitHub. Specifically, we focus on transactional prompts: reproducible natural language instructions that are integrated into software. To enable the empirical, quantitative study of prompts, we introduce a structured ontology, capturing the properties of prompts as well as their formal and semantic components. Based on this ontology, we transform prompts from unstructured raw texts into richly structured linguistic objects. Analysis of these structured data reveals significant diversity of usage patterns across languages, domains, tasks, and modalities, in a typical Zipf-like distribution where some clearly prevail and others, more diverse, appear in the long tail. To validate the reliability of the ontology-based annotation of the prompts, we perform a comprehensive error analysis across all fields, providing a detailed assessment of annotation quality. We release the dataset together with a browsing and exploration interface (https://github.com/OnlpLab/transactionalPromptsCollection ).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。