用多阶段方法从生活日志中精准检索图像。
LifeIR at the NTCIR-18 Lifelog-6 Task
- 分阶段处理：去模糊、重写查询、扩展候选集、多模态模型重排。
- 在NTCIR-18挑战中各阶段均提升检索效果，整体表现优异。
- 适合需要从穿戴设备日志中找特定场景图像的研究者。
近年来，通过运动手表、GoPro等可穿戴设备记录的生活日志分享日益流行。生活日志包含图像、视频和GPS数据等多种信息，反映用户的生活方式、饮食习惯与身体活动。NTCIR-18 Lifelog-6挑战中的生活日志语义访问任务（LSAT）旨在根据描述动作或事件的文本查询，从大规模用户生活日志中检索相关图像，满足用户回溯历史场景的需求。我们提出一种多阶段检索管道，应对生活日志检索中的多重挑战。该管道包括：过滤模糊图像，重写查询以更清晰表达意图，基于事件扩展候选集以纳入时间关联图像，并使用具备更强相关性判断能力的多模态大语言模型（MLLM）进行结果重排。提交结果表明，各阶段及整个管道均有效提升了检索性能。
原文摘要 · Abstract (English)
In recent years, sharing lifelogs recorded through wearable devices such as sports watches and GoPros, has gained significant popularity. Lifelogs involve various types of information, including images, videos, and GPS data, revealing users' lifestyles, dietary patterns, and physical activities. The Lifelog Semantic Access Task(LSAT) in the NTCIR-18 Lifelog-6 Challenge focuses on retrieving relevant images from a large scale of users' lifelogs based on textual queries describing an action or event. It serves users' need to find images about a scenario in the historical moments of their lifelogs. We propose a multi-stage pipeline for this task of searching images with texts, addressing various challenges in lifelog retrieval. Our pipeline includes: filtering blurred images, rewriting queries to make intents clearer, extending the candidate set based on events to include images with temporal connections, and reranking results using a multimodal large language model(MLLM) with stronger relevance judgment capabilities. The evaluation results of our submissions have shown the effectiveness of each stage and the entire pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。