arXiv:2507.03620cs.SEcs.AI2025-07被引 3

用代码化方法优化提示词,让大模型更准更省

Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy

  • 将提示词当作可编程代码,自动优化指令与示例
  • 提示词评估任务准确率从46.2%提升至64.0%
  • 适合想系统提升大模型效果的开发者和研究者

尽管提示工程是释放大语言模型潜力的关键,但设计有效提示仍依赖耗时的人工试错。本研究考察了声明式自优化框架DSPy在五类场景中的应用:护栏执行、代码幻觉检测、代码生成、路由代理和提示评估。结果表明,部分任务有显著提升:提示评估任务准确率从46.2%升至64.0%;路由代理任务中,优化后提示使准确率从85.0%提升至90.0%。但使用低成本模型配合优化提示并未带来性能提升。研究显示,当指令调优与示例选择协同优化时,DSPy能有效提升模型表现,但效果因任务而异,强调需针对具体场景评估提示优化价值。

原文摘要 · Abstract (English)

Although prompt engineering is central to unlocking the full potential of Large Language Models (LLMs), crafting effective prompts remains a time-consuming trial-and-error process that relies on human intuition. This study investigates Declarative Self-improving Python (DSPy), an optimization framework that programmatically creates and refines prompts, applied to five use cases: guardrail enforcement, hallucination detection in code, code generation, routing agents, and prompt evaluation. Each use case explores how prompt optimization via DSPy influences performance. While some cases demonstrated modest improvements - such as minor gains in the guardrails use case and selective enhancements in hallucination detection - others showed notable benefits. The prompt evaluation criterion task demonstrated a substantial performance increase, rising accuracy from 46.2% to 64.0%. In the router agent case, the possibility of improving a poorly performing prompt and of a smaller model matching a stronger one through optimized prompting was explored. Although prompt refinement increased accuracy from 85.0% to 90.0%, using the optimized prompt with a cheaper model did not improve performance. Overall, this study's findings suggest that DSPy's systematic prompt optimization can enhance LLM performance, particularly when instruction tuning and example selection are optimized together. However, the impact varies by task, highlighting the importance of evaluating specific use cases in prompt optimization research.

提示优化DSPy大模型自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。