研究大模型在选择题中角色生成的不稳定性，发现不同模型和题目类型差异显著。
Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions
- 设计三类指标，从性能、结果、题目正确性三方面衡量角色生成稳定性
- 数学与常识类题目导致的不稳定性最高，模型规模和家族影响明显
- 提示格式比温度等超参更易引发不稳定，需关注实验设置对角色表现的影响
角色驱动生成（PDG）在科研与工业应用中广泛使用，大语言模型（LLM）在任务中扮演特定角色。尽管自由文本形式的角色表达已有较多关于稳定性的研究，但非文本密集型输出（如多选题问答，MCQA）中的角色表达却常被忽视。本文填补该空白，探究LLM在MCQA任务中角色生成的不稳定性。我们提出三个指标，分别评估性能、结果和题目正确性在三个维度上的稳定性。通过这些指标发现，不稳定性在不同模型家族和模型规模间存在系统性差异，且在数学与常识类问题上尤为突出。此外，任务提示格式带来的预测不稳定性高于温度等其他超参数。最后，我们发现不稳定性与任务准确率相关，并通过不稳定性指标识别出不同实验设置下表现最佳与最差的角色，即使角色本身相似。这凸显了在角色生成中检查超参数不稳定性的重要性。
原文摘要 · Abstract (English)
Persona-driven generations (PDGs) have seen prolific use in research and industry applications, where a large language model (LLM) takes on a 'persona' while completing some task. While persona expressed through free-form text (like dialogue) has substantial work investigating stability or consistency, relatively, persona expressed in non-text-heavy outputs (like in multiple-choice question answering, or MCQA) is often overlooked. We work to address this gap, seeking to understand the instability of LLM PDGs in MCQA tasks. We develop three metrics investigating the performance, outcome, and question correctness stability, evaluating three distinct dimensions. Using these metrics, we find that instability varies consistently between model families and model size, and across question domains, with math/commonsense questions leading to greater instability. We also find task prompt format introduces more prediction instability than other hyperparameters, like temperature. Finally, we find that instability is related to task accuracy, and using our instability metrics, find different experimental settings that result in different best and worst personas for tasks, despite their similarity. This reveals the importance of checking hyperparameter instability in PDGs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。