让大模型回答更公平,避免不同问法导致答案质量差异。
Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO

- 用加权多提示池选择策略,最小化最差情况下的回答质量损失。
- 在双语医疗与金融数据集上,最差25%回答质量提升13.2%。
- 适合需要高公平性、多场景部署的医疗/金融类AI应用。
大型语言模型在信息查询中日益普及,但语义相同的提问因表达方式不同,可能获得质量差异显著的回答。系统提示被广泛用于引导响应行为,但通常仅针对平均表现优化,导致某些提问方式仍会得到不完整或低质量的答案。为此,本文提出一种受限混合策略分组DRO(Constrained Mixed-Strategy GroupDRO)框架,用于系统提示选择。该框架不优化提示文本本身,而是对现有提示池中的提示分配权重,以最小化在评估指标和用户群体上的最差情况信息质量损失,同时约束平均损失接近基于平均选择的结果。由于提示生成与选择解耦,该方法可适配任意提示池,并能利用互补提示的集合而非单一提示。在五个LLM、两个双语医学与消费者金融基准上的实验表明,该方法相比无缓解措施,平均将总体均值、最差25%均值和最差值分别降低13.1%、13.2%和13.7%,同时保持总体质量与平均选择相近。多提示权重揭示了不同指标-群体组合间的互补性。代码与数据已公开于https://github.com/Rainxu09/equitable-system-prompt-selection。
原文摘要 · Abstract (English)
Large language models are increasingly used for information seeking, yet semantically equivalent questions phrased in different ways can receive answers of considerably different quality. System prompts are widely employed to steer response behavior, but they are typically optimized for average-case quality, so some question phrasings may still receive incomplete or low-quality answers. To address this, we formulate a constrained mixed-strategy GroupDRO framework for system-prompt selection. Instead of optimizing the system-prompt text, the framework assigns weights to system prompts in an existing pool to minimize the worst-case information-quality loss across evaluation metrics and groups, while constraining the mean loss to stay close to that of average-based selection. Because pool generation and selection are decoupled, the method applies to any system-prompt pool and can leverage an ensemble of complementary system prompts rather than a single one. Across five LLMs on two bilingual medical and consumer-finance benchmarks, the constrained method reduces the Overall Mean, Worst 25% Mean, and Worst by 13.1%, 13.2%, and 13.7% on average relative to no mitigation while keeping overall quality close to Average selection. Its multi-prompt weights reveal complementarity across metric-group pairs. Code and data are available at https://github.com/Rainxu09/equitable-system-prompt-selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。