让提示词在准确率和计算成本间自由权衡,不再预设取舍。
CRAFT: Cost-aware Refinement And Front-aware Tuning of Prompts

- 基于帕累托前沿优化提示词,动态平衡准确率与成本
- 在6个任务中同时覆盖高准确率和低成本区域
- 适合需要灵活部署的模型应用开发者
为提升准确率而优化的提示词通常变长,增加每次调用模型的推理成本。最优的准确率-成本权衡取决于任务和预算,因此提示词优化应是在准确率与提示词令牌成本构成的帕累托前沿上进行搜索,而非寻找单一最优提示。传统方法将目标合并为加权和,预先固定权衡系数,常导致仅能捕捉前沿的一小部分,称为标量化解耦。本文提出CRAFT(Cost-aware Refinement And Front-aware Tuning),一种面向帕累托前沿的提示词优化器,将目标大语言模型验证调用视为稀缺资源,分配给接近乐观前沿的候选提示。每轮迭代中,分别侧重准确率和成本的生成器提出修改建议,帕累托间隙获取策略使用每轮预算进行验证,NSGA-II保留策略维持种群多样性。在六个分类与推理基准测试中,CRAFT保留的前沿同时覆盖了高准确率与低成本区域,而仅追求准确率、仅追求成本或加权和基线均集中于较窄区域。准确率-成本权衡变为搜索后的选择,而非搜索前的预设参数。
原文摘要 · Abstract (English)
Prompts tuned for accuracy often grow long, raising inference cost on every model call. The best accuracy-cost trade-off depends on the task and the budget, so prompt optimization is a search over the Pareto front of accuracy and prompt-token cost rather than for one prompt. The usual shortcut, collapsing the objectives into a weighted sum, fixes the trade-off weight before search and often recovers only a narrow region of the front, a failure we call scalarization collapse. We present CRAFT (Cost-aware Refinement And Front-aware Tuning), a Pareto-front prompt optimizer that treats target-LLM validation calls as the scarce resource and allocates them to candidates near the optimistic candidate front. Each round, complementary accuracy-oriented and cost-oriented generators propose edits, Pareto-gap acquisition spends the per-round validation budget, and NSGA-II retention keeps a spread-out population. Across six classification and reasoning benchmarks, CRAFT's retained fronts reach both high-accuracy and low-cost regions, while accuracy-only, cost-only, and weighted-sum baselines each concentrate in narrower regions. The accuracy-cost trade-off becomes a post-search choice, not a pre-search weight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。