测试代码大模型对情绪化提示的敏感度,发现性能与稳定性常不一致。
Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variations
- 用情绪和人格模板生成语义相同但风格不同的提示变体。
- 14个模型在不同提示下表现差异大,稳定性与规模/架构相关。
- 适合关注模型可靠性与部署风险的开发者和研究者。
代码生成模型广泛用于软件开发,但其对提示措辞的敏感性尚未深入研究。相同需求以不同情绪或沟通风格表达时,可能产生显著不同的输出,而现有评测多仅关注峰值性能。本文提出 PromptSE(提示敏感性评估)框架,通过情绪与人格模板生成语义等价的提示变体,并采用概率感知连续评分或二值通过率(当logits不可用时)评估稳定性。结果汇总为提出的曲线下面积指标(AUC-E),实现跨模型比较。在来自Llama、Qwen、DeepSeek三个系列的14个模型上,研究发现性能与稳定性作为优化目标基本解耦,并揭示了挑战常见鲁棒性假设的架构与规模模式。该框架支持闭源模型快速筛选及研究场景下的详细稳定性分析。PromptSE使从业者可量化评估部署中性能与稳定性的权衡,将提示稳定性定位为继性能与公平性之后的补充评价维度,助力构建更可信的AI辅助开发工具。
原文摘要 · Abstract (English)
Code generation models are widely used in software development, yet their sensitivity to prompt phrasing remains under-examined. Identical requirements expressed with different emotions or communication styles can yield divergent outputs, while most benchmarks emphasize only peak performance. We present PromptSE (Prompt Sensitivity Evaluation), a framework that creates semantically equivalent prompt variants with emotion and personality templates, and that evaluates stability using probability aware continuous scoring or using binary pass rates when logits are unavailable. The results are aggregated into a proposed area under curve metric (AUC-E) for cross model comparison. Across 14 models from three families (Llama, Qwen, and DeepSeek), our study shows that performance and stability behave as largely decoupled optimization objectives, and it reveals architectural and scale related patterns that challenge common assumptions about model robustness. The framework supports rapid screening for closed-source models as well as detailed stability analysis in research settings. PromptSE enables practitioners to quantify performance stability trade offs for deployment and model selection, positioning prompt stability as a complementary evaluation dimension alongside performance and fairness, and contributing to more trustworthy AI-assisted software development tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。