用多元用户角色评估生成式界面,更贴近真实用户反馈。
Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation

- 构建心理多样化的虚拟用户群,基于证据打分并交互讨论。
- 相关性提升至0.922,显著优于单一模型评判和提示集成方法。
- 可发现不同用户群体在界面评价上的分歧,适合产品优化参考。
生成式界面(GenUI)使大语言模型能直接从自然语言指令生成完整可渲染的界面,但其生成质量评估仍属难题。人工评估成本高且主观性强,而以大模型为评判者虽可扩展,却仅反映单一隐含视角,难以捕捉真实用户群体对同一界面的不同感知。本文提出证据锚定、社会加权的用户画像评估框架(ESPP),包含三阶段:一组心理特征多样、基于证据的虚拟用户独立评分;在基于特质的语义受限信任机制下交换意见;最后通过受德尔菲法启发的社会加权聚合为最终判断。相比单次评判,该方法将人类判断相关性从0.716提升至0.922,提示集成仅恢复其中约三分之一差距,证明用户画像与证据锚定是主要改进来源。此外,保留各成员独立评分发现,不同用户子群对整体模型排名一致,但在具体维度上分歧显著,这是单一同质化评判者会系统性掩盖的结构性差异。代码已开源:https://github.com/Wuzheng02/ESPP。
原文摘要 · Abstract (English)
Generative UI (GenUI) lets large language models synthesize a complete, renderable interface directly from a natural-language instruction, but evaluating the quality of what they generate remains an open problem. Human evaluation is costly and rater-variant, while LLM-as-a-judge is scalable but reflects only a single implicit viewpoint, unable to capture how different populations of real users actually perceive the same interface. We propose the Evidence-Grounded, Social-Weighted Persona Panel (ESPP), a three-stage GenUI evaluation method in which a panel of psychologically diverse, evidence-grounded personas independently rates a screenshot, exchanges opinions under a trait-derived, semantically-gated bounded-confidence mechanism, and is aggregated via Delphi-inspired social weighting into a single judgment. ESPP tracks human judgment substantially more closely than a naive single-pass judge, raising Pearson $r$ from $0.716$ to $0.922$, and a prompt-ensemble control recovers only about a third of this gap, isolating genuine persona and evidence grounding as the dominant source of improvement. Beyond this fidelity gain, retaining each panelist's individual rating further reveals that user subgroups agree on overall model rankings yet diverge sharply on specific rating dimensions, a structural disagreement a single homogeneous judge would systematically erase. The codes are available at https://github.com/Wuzheng02/ESPP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。