用数字孪生生成带真实标签的手机助手评测数据，保护隐私且结果可复现。
SenWorld: A Digital-Twin Simulation for Generating Context-Rich Evaluation Data
- 构建物理真实的数字孪生环境，通过事件溯源生成带固定标签的数据
- 生成数据与真实用户数据在分布和通信节奏上高度一致（JSD<0.1）
- 适用于评估手机助手的隐私敏感功能，如通话与短信，无需人工标注
智能手机个人助手需处理长期个人数据，但评估需上下文丰富的评测数据且答案已知，而真实设备日志因隐私问题难以共享。为此，我们提出SenWorld——一个基于真实地图、天气、节假日和网络数据构建的物理可信、确定性、事件溯源的数字孪生仿真系统，生成具有构造性真实标签的数据。在该系统中，16个角色经历完整一天，所有可观测信号均存入全系统快照；每个评估案例通过指向已有记录的指针标注，而非事后标注或大模型判断。在对北京16名角色的测试中，生成数据在类别分布（Jensen-Shannon散度0.070）和每日通信节奏（JSD低于0.1）上接近真实基准，尽管生成记录较短。未预设交互下，角色形成互惠对话子图和差异化行为模式。将数据投射至717个评估案例，暴露了生产级助手78处故障，集中于通话与短信记录，而联系人、日程和闹钟从未失败。快照指针确认均为助手端检索错误，无大模型参与。总体而言，SenWorld提供了隐私安全、可复现且分布可验证的评测数据生成路径。
原文摘要 · Abstract (English)
Smartphone personal assistants reason over longitudinal personal data, yet evaluating them requires context-rich evaluation data whose correct answers are known, and real device traces are too privacy-sensitive to share. To address this challenge, we present SenWorld, a physically grounded, deterministic, event-sourced digital-twin simulation that generates such data with ground truth fixed by construction. In SenWorld, personas live through a full day in a world built from real map, weather, holiday, and network data; every observable signal is archived in full-system snapshots; and each evaluation case is labeled by a pointer to an existing record rather than by post-hoc annotation or a large language model (LLM) judge. We evaluate this method with 16 personas in Beijing. The generated data closely matches the held-out real-user benchmark in category distribution (Jensen--Shannon divergence (JSD) 0.070) and in the daily rhythm of communication records (JSD below 0.1), though generated records remain shorter than real ones. Without scripted interaction, personas form a fully reciprocated dialogue subgraph and differentiated behavioral repertoires. Projected into 717 evaluation cases, the generated data exposes 78 failures in a production smartphone assistant, concentrating on call and Short Message Service (SMS) records while contacts, schedules, and alarms never fail. The snapshot pointer confirms each failure as an assistant-side retrieval error, with no LLM judge involved. Overall, SenWorld offers a privacy-safe, reproducible, and distribution-checked path to evaluation data whose labels are fixed by construction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。