用户解读主题时受认知偏差影响，需建立更真实评估体系
Objectifying the Subjective: Cognitive Biases in Topic Interpretations
- 通过用户研究发现主题解读依赖显著词与代表性启发法
- 提出基于锚定调整理论的解释模型，揭示判断过程机制
- 适合关注人机交互、主题建模评估的学者与工程师
主题解读对下游应用至关重要。现有主题质量评估方法如主题连贯性与词插入测试，未能衡量主题在促进语料探索中的实际作用。为设计基于任务和用户群体的评估体系，我们开展用户研究，了解用户如何理解主题。提出主题质量的构念，并请用户在具体主题情境中进行评估并说明理由。采用反思性主题分析法，从评估理由中识别出主题解读的典型模式。结果显示，用户依据可及性与代表性启发法进行解读，而非概率逻辑。据此提出基于锚定-调整启发式的主题解释理论：用户以显著词为锚点，通过语义调整形成整体理解。主题解读可视为生态理性用户在不确定性下的判断行为，因此需要引入认知偏差意识的用户模型与评估框架。
原文摘要 · Abstract (English)
Interpretation of topics is crucial for their downstream applications. State-of-the-art evaluation measures of topic quality such as coherence and word intrusion do not measure how much a topic facilitates the exploration of a corpus. To design evaluation measures grounded on a task, and a population of users, we do user studies to understand how users interpret topics. We propose constructs of topic quality and ask users to assess them in the context of a topic and provide rationale behind evaluations. We use reflexive thematic analysis to identify themes of topic interpretations from rationales. Users interpret topics based on availability and representativeness heuristics rather than probability. We propose a theory of topic interpretation based on the anchoring-and-adjustment heuristic: users anchor on salient words and make semantic adjustments to arrive at an interpretation. Topic interpretation can be viewed as making a judgment under uncertainty by an ecologically rational user, and hence cognitive biases aware user models and evaluation frameworks are needed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。