测试大模型在社会模拟中的可靠性，发现表现不稳且与通用能力无关。
Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?
- 构建涵盖10个社会学主题的评估数据集TrustSim
- 14个大模型在角色模拟中普遍存在不一致行为
- 提出AdaORPO算法提升7个模型的模拟可靠性
大型语言模型（LLMs）在社会模拟中的应用日益广泛，涉及角色扮演代理和计算社会科学（CSS）。然而，这些模拟的可靠性尚未充分探索，引发对其可信度的担忧。本文旨在回答“基于LLM的模拟有多可靠？”为此，我们引入TrustSim评估数据集，覆盖10个与计算社会科学相关的主题，系统性地研究了基于LLM模拟的可靠性。我们在14个主流大模型上进行了实验，发现模拟角色存在持续不一致的问题。此外，模型的一致性水平与其通用性能之间无强相关性。为提升模拟可靠性，我们提出了基于自适应学习率的强化学习算法AdaORPO，该方法在7个大模型上有效提升了模拟稳定性。本研究为未来构建更稳健、可信赖的基于大模型的社会模拟奠定了基础。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly employed for simulations, enabling applications in role-playing agents and Computational Social Science (CSS). However, the reliability of these simulations is under-explored, which raises concerns about the trustworthiness of LLMs in these applications. In this paper, we aim to answer ``How reliable is LLM-based simulation?'' To address this, we introduce TrustSim, an evaluation dataset covering 10 CSS-related topics, to systematically investigate the reliability of the LLM simulation. We conducted experiments on 14 LLMs and found that inconsistencies persist in the LLM-based simulated roles. In addition, the consistency level of LLMs does not strongly correlate with their general performance. To enhance the reliability of LLMs in simulation, we proposed Adaptive Learning Rate Based ORPO (AdaORPO), a reinforcement learning-based algorithm to improve the reliability in simulation across 7 LLMs. Our research provides a foundation for future studies to explore more robust and trustworthy LLM-based simulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。