用关键事件分析法解析大模型性格差异。
Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis
- 构建涵盖多领域的事件数据集，评估模型提取关键事件能力。
- 不同模型展现明显风格差异：情感、战略、实证、因果推理等倾向。
- 通过另一模型评判，提升大模型决策可解释性，适合应用开发与评测。
大型语言模型（LLMs）日益融入日常应用，其决策机制与内在人格的理解变得至关重要。本文提出「超新星事件数据集」（Supernova Event Dataset），包含传记、历史事件、新闻和科学发现等多样文本。该数据集用于评估模型在长文本中提取并排序关键事件的能力，这是一项需处理长程上下文与因果链的主观复杂任务。我们测试了小模型如Phi-4、Orca 2、Qwen 2.5，以及大模型如Claude 3.7、Gemini 2.5、OpenAI o3。提出一个框架：由另一个大模型充当裁判，基于模型对事件的选择与分类推断其人格特征。分析显示显著性格差异：Orca 2 倾向于情感化人际分析，Qwen 2.5 展现策略性分析风格；在科学发现事件中，Claude Sonnet 3.7 注重概念建构，Gemini 2.5 Pro 强调实证验证，o3 则偏好分步因果推理。该方法增强了模型可解释性，有助于多样化应用场景的用户友好设计。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly integrated into everyday applications. As their influence grows, understanding their decision making and underlying personality becomes essential. In this work, we interpret model personality using our proposed Supernova Event Dataset, a novel dataset with diverse articles spanning biographies, historical events, news, and scientific discoveries. We use this dataset to benchmark LLMs on extracting and ranking key events from text, a subjective and complex challenge that requires reasoning over long-range context and modeling causal chains. We evaluate small models like Phi-4, Orca 2, and Qwen 2.5, and large, stronger models such as Claude 3.7, Gemini 2.5, and OpenAI o3, and propose a framework where another LLM acts as a judge to infer each model's personality based on its selection and classification of events. Our analysis shows distinct personality traits: for instance, Orca 2 demonstrates emotional reasoning focusing on interpersonal dynamics, while Qwen 2.5 displays a more strategic, analytical style. When analyzing scientific discovery events, Claude Sonnet 3.7 emphasizes conceptual framing, Gemini 2.5 Pro prioritizes empirical validation, and o3 favors step-by-step causal reasoning. This analysis improves model interpretability, making them user-friendly for a wide range of diverse applications. Project Page - https://www.supernova-event.ai/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。