用游戏化任务量化人与AI的隐性理解能力
TUX: Measuring Human--AI Tacit Understanding

- 设计类
- 241人+200个模型测试,发现相似人格者更易达成隐性对齐
- 适合研究人机协作、心理建模或个性化交互的学者
随着大语言模型日益成为协作伙伴,人类-人工智能对齐通常通过明确任务成功率、准确率或奖励优化来评估。然而许多协作场景依赖于隐性理解:即代理能否在缺乏明确目标、沟通或反馈的情况下,与人类的评价立场或表征先验保持一致。为研究这一能力,我们设计了一种受社交游戏Wavelength启发的谱位放置任务,让人类与代理独立将概念置于主观光谱上。我们提出隐性理解指数(TUX),作为衡量人类与代理判断相似性的成对指标,并在241名参与者和200个基于个人资料条件化的LLM代理(覆盖四个模型)上进行评估。结果发现,特质空间中最近的人类-代理配对显著获得更高TUX,表明隐性对齐由个体特征决定而非随机相似。回归分析显示,随着预测变量集丰富度提升,TUX的可解释性增强,个体特质、决策风格和信心表现优于聚合特质距离基线。这些发现表明人类与大模型间的隐性理解可被测量,同时揭示了基于个人资料条件化在捕捉深层表征对齐上的局限。
原文摘要 · Abstract (English)
As large language models (LLMs) increasingly act as collaborative partners, human--AI alignment is often evaluated through explicit task success, accuracy, or reward optimization. Yet many collaborative settings depend on tacit understanding: whether an agent can align with a human's evaluative stance or representational priors without clear objectives, communication, or feedback. To study this capacity, we develop a spectrum-placement task inspired by the social party game Wavelength, in which humans and agents independently place concepts along subjective spectra. We operationalize the Tacit Understanding Index (TUX) as a pairwise measure of similarity between human and agent judgments, and evaluate it with 241 human participants and 200 profile-conditioned LLM agents across four models. We find that nearest human--agent pairs in trait space achieve significantly higher TUX, suggesting that tacit alignment is structured by person-level characteristics rather than random similarity. Regression analyses show that TUX becomes more explainable as predictor sets become richer, with individual traits, decision-making styles, and confidence improving over aggregate trait-distance baselines. These findings suggest that tacit understanding between humans and LLMs is measurable, while revealing the limits of profile-based conditioning for capturing deeper representational alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。