用大模型模拟用户体验，低成本验证产品设计早期决策。
Proxy-Validated LLM UX Micro-Simulations: An Artifact-First Protocol for Early-Stage Decision Support
- 通过版本化提示词和角色生成用户操作路径与痛点反馈。
- 利用应用评论等公开数据验证模拟结果，嵌入向量匹配效果优于词法方法。
- 提供可复现的流水线工具，适合产品团队早期快速试错。
早期团队常因缺乏用户、时间与预算，无法开展多次用户体验研究，但仍需决策信号以安全迭代。本文提出一种基于大模型的用户体验微仿真流程，从版本化提示词、角色设定、任务指令和界面快照中生成结构化用户反馈（如操作步骤、摩擦点、微型调查信号）。由于带有任务结果的公开可用性数据集稀缺，我们使用多个公开代理语料库（应用评论、支持推文、开源软件问题）对模拟出的摩擦主题进行验证。提出轻量级代理验证协议，包含两个对齐度量：top-k Jaccard 和分布加权Jaccard（W），并在六个代理数据集上对比了词法、TF-IDF 和多语言嵌入基线。结果显示，嵌入方法在主要应用评论与支持推文代理上显著优于词法基线（如Gojek上W=0.128 vs 0.000），而top-k Jaccard在k值较大时高估对齐程度。我们在Azure OpenAI部署中评估四种智能体策略（单次遍历、最佳N个、混合、及新提出的评分后选择判别器），并报告8组方法-数据集组合的自举置信区间；结果显示，在当前子样本规模下，嵌入W的点估计系统性不稳定。此外，还分析了接地与虚构类代理的失败模式，记录校准注意事项，并给出被对抗判别器标记为虚构输出的示例。本成果采用首件优先的流水线，从版本化运行产物生成可复现的表格与图表，支持在最终付费模型校准前持续优化提示词与分类体系。
原文摘要 · Abstract (English)
Early-stage teams often lack users, time, and budget to run repeated UX studies, yet still need decision-oriented signals to iterate safely. We study an LLM-driven UX micro-simulation pipeline that generates structured customer-experience feedback (walkthrough steps, friction points, micro-survey signals) from versioned prompts, personas, tasks, and UI snapshots. Because public usability datasets with task outcomes are scarce, we validate simulated friction themes using multiple public proxy corpora (app reviews, support tweets, and open-source software issues). We propose a lightweight proxy-validation protocol with two alignment metrics: top-k Jaccard and distributional weighted-Jaccard (W), and compare lexical, TF-IDF, and multilingual embedding baselines across six proxy datasets. Embedding-based alignment yields higher W than lexical baselines on primary app-review and support-tweet proxies (e.g., W=0.128 vs 0.000 on Gojek), while top-k Jaccard is shown to overstate alignment at large k. We ablate four agent strategies (single-pass, best-of-N, hybrid, and a proposed score-then-select judge) across Azure OpenAI deployments and report bootstrap confidence intervals over 8 method-dataset pairs; these intervals reveal that the embedding W point estimate is systematically unstable under resampling at our subsample size. We also provide a failure-mode analysis of grounding and fabrication proxies, with documented calibration caveats and worked examples of outputs flagged as fabricated by an adversarial judge. Our artifact-first pipeline produces reproducible tables and figures from versioned run artifacts, supporting iterative prompt and taxonomy refinement before final paid-model calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。