测试大模型如何模拟作者思维,发现结合概念与语言特征最有效
Individualized Cognitive Simulation in Large Language Models: Evaluating Different Cognitive Representation Methods
- 构建新任务与11条件评估框架,对比七种认知表征方法
- 融合概念与语言特征的表征效果最佳,优于静态人物画像
- 大模型更擅长模仿语言风格,难还原叙事结构,揭示深层模拟局限
个体化认知模拟(ICS)旨在构建逼近特定个体思维过程的计算模型。尽管大语言模型(LLMs)能逼真模仿角色扮演等表面行为,但其在模拟深层个体认知方面的能力仍不清晰。为此,我们提出一项新任务,评估不同认知表征方法在ICS中的表现。基于发布于所测试LLM之后的近期小说构建数据集,并设计11条件认知评估框架,对七款现成LLMs在作者风格模仿任务中进行基准测试。假设有效的认知表征可使LLM生成更贴近原作者的叙事内容,因此测试了语言特征、概念映射及基于个人资料的信息等不同表征方式。结果显示,结合概念与语言特征的表征在整体评估中表现最优,优于静态的个人资料提示。重要的是,LLMs在模仿语言风格方面优于叙事结构,凸显其在深层认知模拟上的局限性。这些发现为开发适应个体思维与表达方式的AI系统奠定基础,推动更个性化、更符合人类意图的创作技术发展。
原文摘要 · Abstract (English)
Individualized cognitive simulation (ICS) aims to build computational models that approximate the thought processes of specific individuals. While large language models (LLMs) convincingly mimic surface-level human behavior such as role-play, their ability to simulate deeper individualized cognitive processes remains poorly understood. To address this gap, we introduce a novel task that evaluates different cognitive representation methods in ICS. We construct a dataset from recently published novels (later than the release date of the tested LLMs) and propose an 11-condition cognitive evaluation framework to benchmark seven off-the-shelf LLMs in the context of authorial style emulation. We hypothesize that effective cognitive representations can help LLMs generate storytelling that better mirrors the original author. Thus, we test different cognitive representations, e.g., linguistic features, concept mappings, and profile-based information. Results show that combining conceptual and linguistic features is particularly effective in ICS, outperforming static profile-based cues in overall evaluation. Importantly, LLMs are more effective at mimicking linguistic style than narrative structure, underscoring their limits in deeper cognitive simulation. These findings provide a foundation for developing AI systems that adapt to individual ways of thinking and expression, advancing more personalized and human-aligned creative technologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。