用认知模型改进GPT-4文本相似度,更贴近人的真实判断。
Leveraging a Cognitive Model to Measure Subjective Similarity of Human and GPT-4 Written Content
- 结合认知模型与LLM嵌入,构建个性化相似度度量。
- 在钓鱼邮件分类任务中,新方法更符合人类主观判断。
- 适合教育和推荐场景中需考虑个体偏见的场景。
使用GPT-4等大语言模型的词元嵌入计算文档间的余弦相似度,可广泛应用于文档分类等任务。然而,这种相似度依赖于模型训练语料,未必反映个体主观感知或其偏见与限制对相似性判断的影响。在教育与推荐系统中,因个体判断数据有限且偏见影响显著,这一问题尤为突出。为此,本文融合实例学习(IBL)认知模型与LLM嵌入,提出实例化个体化相似度(IBIS)度量。该方法基于决策的认知机制,能有效体现个体偏见与约束。为验证效果,我们构建了一个包含人类对邮件是否为钓鱼邮件(phishing)或安全邮件(ham)分类的数据集,结果表明,引入认知模型可显著提升相似度度量与人类主观判断的一致性。
原文摘要 · Abstract (English)
Cosine similarity between two documents can be computed using token embeddings formed by Large Language Models (LLMs) such as GPT-4, and used to categorize those documents across a range of uses. However, these similarities are ultimately dependent on the corpora used to train these LLMs, and may not reflect subjective similarity of individuals or how their biases and constraints impact similarity metrics. This lack of cognitively-aware personalization of similarity metrics can be particularly problematic in educational and recommendation settings where there is a limited number of individual judgements of category or preference, and biases can be particularly relevant. To address this, we rely on an integration of an Instance-Based Learning (IBL) cognitive model with LLM embeddings to develop the Instance-Based Individualized Similarity (IBIS) metric. This similarity metric is beneficial in that it takes into account individual biases and constraints in a manner that is grounded in the cognitive mechanisms of decision making. To evaluate the IBIS metric, we also introduce a dataset of human categorizations of emails as being either dangerous (phishing) or safe (ham). This dataset is used to demonstrate the benefits of leveraging a cognitive model to measure the subjective similarity of human participants in an educational setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。