构建多模态个人化评测基准,推动模型理解真实生活记忆
Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition
- 设计11800+问答对的合成数据集,按证据范围分三类任务
- 跨任务准确率在聚合推理任务中低于0.40,显示当前方法仍不足
- 提出知识图谱框架LifeGraph,支持按需检索视觉证据
随着大语言模型日益用于个人助手,用户期望其能基于多模态生活史进行推理,从识别人物到理解事件再到归纳模式,但现有评测主要聚焦概念识别。我们提出Life-Bench,一个完全合成且经人工验证的多模态基准,包含超过11,800个问答对,覆盖10项任务,按所需证据范围分为概念识别、事件理解与聚合推理三类。数据集的图像为中心的个人历史在嵌入统计上与真实用户账户分布一致。个人数据的互联结构适合图模型解决;我们提出LifeGraph,一种提供结构化检索并按需访问原始视觉证据的个人知识图谱框架,在事件和聚合任务中表现尤为突出。对四种检索范式的系统评估表明,准确率随证据范围扩大而急剧下降,在聚合任务中低于0.40,且无单一范式在所有类别中占优。所有方法在概念识别之外的表现仍有限,确立了多模态生活史个性化为开放挑战,而Life-Bench则成为未来进展的测试平台。
原文摘要 · Abstract (English)
As large language models increasingly power personal assistants, users expect them to reason over multimodal life histories, from recognizing people to understanding events to aggregating patterns, yet existing benchmarks primarily target concept-level recognition. We introduce Life-Bench, a fully synthetic, human-verified multimodal benchmark of over 11,800 question-answer pairs across 10 tasks, organized by required evidence scope: concept identification, event understanding, and aggregated reasoning. The benchmark's photo-centric personal histories are distributionally aligned with real user accounts under embedding statistic. The interconnected structure of personal data invites graph-based solutions; we propose LifeGraph, a personal knowledge graph framework providing structured retrieval with on-demand access to source visual evidence, showing particular promise on event and aggregated tasks. Systematic evaluation of four retrieval paradigms on Life-Bench demonstrates that accuracy degrades sharply with evidence scope, falling below 0.40 on aggregated tasks, and that no single paradigm dominates across categories. Performance beyond concept recognition remains modest for all evaluated methods, establishing personalization over multimodal histories as an open challenge and Life-Bench as a testbed for future progress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。