用五种优化方法提升大模型提示词，让机器判断更接近人类标准。
A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation
- 在DSPy框架中比较五种提示优化算法
- 优化后提示词在幻觉检测上超越基准方法
- 适合关注大模型评估对齐的研究者
我们主张，声明式自提升Python（DSPy）优化器是一种将大语言模型（LLM）提示词及其评估结果与人类标注对齐的方法。本文在DSPy框架下，对五种远程提示算法——合作提示优化（COPRO）、多阶段指令提示优化（MIPRO）、BootstrapFewShot、带Optuna的BootstrapFewShot以及K近邻少样本——进行了对比分析，考察其与人类评估的对齐能力。以公开基准数据集为例，重点优化提示词，使其在使用LLM作为裁判时的幻觉检测结果与人工标注真值对齐。实验表明，经优化的提示词能超越多种基准方法，实现更优的幻觉检测性能；在这些实验中，某些远程提示器表现优于其他方法。
原文摘要 · Abstract (English)
We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with human evaluations. As a concrete example, we focus on optimizing the prompt to align hallucination detection (using LLM as a judge) to human annotated ground truth labels for a publicly available benchmark dataset. Our experiments demonstrate that optimized prompts can outperform various benchmark methods to detect hallucination, and certain telemprompters outperform the others in at least these experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。