用真实社区反应评估大模型社会语境理解力，发现当前对齐策略仍不足。
Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities
- 基于社区反应语气构建评估框架，捕捉细微语用态度变化
- 实证显示大模型在社区语境模拟上存在持续的“真实性差距”
- 适合关注大模型社会行为对齐的研究者与实践者
大型语言模型（LLMs）正被广泛用作计算社会分析的代理工具，但其忠实呈现人类社群‘厚描述’（Geertz, 1973）的能力仍是关键挑战。现有评估常将社会身份简化为静态标签，忽视真实群体在社会变迁中的动态应对。为此，我们提出CARE（Community-Aware Reaction Evaluation）——一种以反应为中心的评估框架，通过对比大模型生成话语与真实社区对现实新闻事件的即时反应，实现基准测试。该框架通过细粒度刻画语用语气谱系及其背后的态度，并经由人机协同验证，揭示出一个持续存在的‘真实性差距’：仅通过显式社区提示引导大模型，并不能自发提升其模拟保真度。进一步分析还发现前沿模型间存在分歧性行为特征，表明当前对齐策略仍不足以捕捉在线群体的社会语言动态。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly utilized as proxies for computational social analysis; yet, their ability to faithfully represent the "thick descriptions" (Geertz, 1973) of human communities remains a critical challenge. Current evaluations often reduce social identity to static labels, sidelining how real-world groups navigate social shifts. To bridge this gap, we introduce CARE (Community-Aware Reaction Evaluation), a reaction-centered framework that benchmarks LLM-simulated discourse against the authentic, event-contingent responses of distinct communities to real-world news. By characterizing a fine-grained spectrum of illocutionary tones and the underlying attitudes they manifest--validated through human-AI collaboration--our diagnosis reveals a persistent "realism gap": steering LLMs with explicit community prompts fails to inherently improve simulation fidelity. Analysis further identifies divergent behavioral signatures among frontier models, suggesting that current alignment strategies remain insufficient for capturing the sociolinguistic dynamics of online groups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。