arXiv:2605.19798cs.CL2026-05

用大模型生成有信任度差异的多模态交互行为,发现性别提示会强化刻板印象。

Towards Trust Calibration in Socially Interactive Agents: Investigating Gendered Multimodal Behaviors Generation with LLMs

论文配图:Towards Trust Calibration in Socially Interactive Agents: Investigating Gendered Multimodal Behaviors Generation with LLMs
图 1 · 摘自论文原文
  • 设计新方法让大模型按需生成体现能力与善意水平的语音、表情、手势等行为
  • 实验证明生成行为与设定的信任维度一致,且用户能正确感知差异
  • 揭示性别提示导致模型强化男女能力与善意的刻板联想,警示应用风险

随着社交交互智能体(SIAs)日益融入日常生活,将用户对代理的信任校准至其真实能力至关重要。本文研究大语言模型(LLMs)生成反映不同能力与善意水平的多模态行为(言语、语调、手势、面部表情)的能力。提出一种自动生成特定信任特质行为的新方法,迈出实现精细信任校准交互的第一步。通过分析大量由LLM生成的多模态转录数据,发现GPT-5.4可在各模态间生成连贯行为。随机森林特征重要性分析显示,生成行为符合能力与善意的理论预期。但当提示中包含性别信息时,模型倾向于重现社会性别刻板印象:男性代理被赋予高能力,女性代理被赋予高善意。为验证方法有效性,在Prolific上开展用户研究(被试内设计),结果显示参与者能准确感知生成行为中预设的能力与善意水平。

原文摘要 · Abstract (English)

As Socially Interactive Agents (SIAs) become increasingly integrated into daily life, the ability to calibrate user trust to an agent's actual capabilities would help ensure appropriate usage of these agents. In this paper, we explore the capacity of Large Language Models (LLMs) to generate multimodal behaviors (verbal, vocal, gestural, and facial expression modalities) that reflect varying levels of ability and benevolence, two key dimensions of trustworthiness. We propose a novel method for automatically generating behaviors aligned with specific levels of these traits, a first step towards enabling nuanced and trust-calibrated interactions. By analyzing a large dataset of multimodal transcripts generated by LLMs, we demonstrate that GPT-5.4 is able to produce coherent behavior across different modalities (text, intonation, facial expression, and gesture). Using Random Forest feature importance analysis, we show that the generated behaviors align with theoretical expectations for ability and benevolence. However, we also find that when gender is specified in the prompt, LLMs tend to reproduce societal gender stereotypes, associating male agents' behaviors with high ability and female agents' behaviors with high benevolence. To validate our approach, we conducted a user study on Prolific using a within-subjects design. Participants perceived different levels of ability and benevolence in the generated behaviors align with the intended instructions.

多模态生成信任建模大模型伦理性别偏见

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。