arXiv:2410.01708cs.CLcs.SI2024-10被引 1

大模型能理解帖子语义并生成符合社交关系的评论,但靠提示词难实现精准个性化。

Examining the Role of Relationship Alignment in Large Language Models

  • 用真实社交数据测试大模型对人际关系语义的感知能力
  • 不给提示词时,模型生成评论与真人评论对社交关系同样敏感
  • 加入所有关系信息反而降低生成质量,暴露训练数据局限

随着生成式AI在社交场景中的广泛应用,如何在保持准确性与真实性的前提下实现用户个性化成为关键问题。本研究基于公开的Facebook帖子-评论数据集,评估了Llama 3.0(70B)在预测不同评论者与发帖者性别、年龄及亲密度组合下的语义语气,并复现这些差异的能力。研究分为两部分:第一部分评估社会关系类别对语义语气的影响;第二部分比较在输入公共帖子的前提下,由Llama 3.0(70B)生成的评论与人类评论的相似性。结果显示,引入社会关系信息可提升模型对人类评论语义语气的预测能力。然而,即使未在提示中包含社会背景信息,大模型生成的评论与人类评论对社会语境的敏感度相当,表明模型能仅从原始帖子中理解语义。当在提示中完整输入社会关系信息后,生成评论与人类评论的相似性反而下降。这一矛盾现象可能源于大模型训练数据中缺乏社会关系信息。整体表明,大模型具备从原始内容理解语义并作出类人回应的能力,但在仅通过提示词实现个性化方面存在明显局限。

原文摘要 · Abstract (English)

The rapid development and deployment of Generative AI in social settings raise important questions about how to optimally personalize them for users while maintaining accuracy and realism. Based on a Facebook public post-comment dataset, this study evaluates the ability of Llama 3.0 (70B) to predict the semantic tones across different combinations of a commenter's and poster's gender, age, and friendship closeness and to replicate these differences in LLM-generated comments. The study consists of two parts: Part I assesses differences in semantic tones across social relationship categories, and Part II examines the similarity between comments generated by Llama 3.0 (70B) and human comments from Part I given public Facebook posts as input. Part I results show that including social relationship information improves the ability of a model to predict the semantic tone of human comments. However, Part II results show that even without including social context information in the prompt, LLM-generated comments and human comments are equally sensitive to social context, suggesting that LLMs can comprehend semantics from the original post alone. When we include all social relationship information in the prompt, the similarity between human comments and LLM-generated comments decreases. This inconsistency may occur because LLMs did not include social context information as part of their training data. Together these results demonstrate the ability of LLMs to comprehend semantics from the original post and respond similarly to human comments, but also highlights their limitations in generalizing personalized comments through prompting alone.

大模型社交语义个性化生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。