提出四维框架，评估大模型回答的表达方式与文化适配性。
Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims
- 构建四维审计框架，分析回应的文化定位、泛化语言等特征。
- 在376k主观问题上测试三款开源模型，发现各特征使用频率差异显著。
- 揭示文化定位与拟人化正相关，适合关注模型沟通风格的研究者使用。
大型语言模型（LLMs）越来越多地用于回答主观性、信息类问题，用户不仅关心答案是否正确，更关注回答的表达方式。现有评估主要聚焦事实正确性，忽视了回应的表述框架。为此，我们提出FRANZ——一个用于响应特征刻画的自动化框架，从文化定位、泛化语言使用、拟人化线索和对话准则遵守四个维度对LLM回应进行沟通审计。为支持该评估，我们构建了SQUARE数据集，包含376,000条来自57个子版块的主观问题，映射至7个国家和19个问题类别。通过评分三款开源大模型的回应，我们发现各模型在不同特征的使用频率上存在统计显著差异。不同于单一维度审计，FRANZ揭示了内部文化定位与拟人化呈现呈正相关，且耦合程度随国家而异，为识别响应框架差异提供了诊断视角。
原文摘要 · Abstract (English)
Large language models (LLMs) are being increasingly used to answer subjective, information-seeking questions, where users are sensitive to how responses are communicated, not just whether the answers are correct. Existing LLM evaluations for subjective cultural queries largely focus on factual correctness, ignoring how the response is framed. To this end, we introduce FRANZ, an automated FRAmework for respoNse characteriZation to conduct communicative audit of LLM responses along four dimensions: cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims. To enable this evaluation, we contribute SQUARE - a corpus of 376k subjective questions sourced from 57 subreddits, and mapped to 7 countries and 19 question categories. We demonstrate FRANZ's applicability by scoring responses from three open-weight LLMs. We observe that LLMs show statistically significant differences in the frequency with which they employ each response characteristic. Unlike single-dimensional audits, FRANZ reveals that insider positioning and anthropomorphism are positively coupled, with the degree of coupling varying by country, providing a diagnostic lens for identifying framing divergences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。