用子模函数优化提示词，提升自动提示工程效果
Submodular Evaluation Subset Selection in Automatic Prompt Optimization
- 基于子模性设计评估子集选择方法，理论可保证性能
- 在GSM8K等数据集上优于随机和启发式基线
- 适合需要高效自动调优的NLP研究者
自动提示优化减少了人工设计提示的工作量，但其反馈信号通常依赖于小规模、随机采样的评估子集。然而，评估子集的选择常被视为实现细节。本文从理论上研究提示优化中的评估子集选择问题，提出SESS方法：将选择过程建模为最大化目标集合函数，在温和条件下该函数具有单调性和子模性，从而支持贪婪选择并提供理论保障。在GSM8K、MATH和GPQA-Diamond三个数据集上，子模选择的评估子集能生成比随机或启发式基线更优的优化提示。
原文摘要 · Abstract (English)
Automatic prompt optimization reduces manual prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized prompts than random or heuristic baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。