WISE让非专业人士也能用自然语言搜索视频图像中的场景、物体、人脸等。
WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata
- 基于向量搜索，支持跨模态混合查询
- 可检索百万级图像或数千小时视频，响应迅速
- 适合档案馆、媒体机构处理敏感数据
本文介绍 WISE，一个开源的音视频多模态搜索引擎，将多种检索功能集成于单一实用工具中，无需机器学习知识即可使用。WISE 支持场景级（如空街）和物体级（如马）的自然语言与反向图像查询；基于人脸搜索特定人物；通过文本（如木头吱呀声）或音频文件检索声音事件；对自动转录的语音内容进行搜索；以及按用户提供的元数据过滤。通过组合多模态查询，可获得丰富洞察——例如，在历史档案中检索德国火车，只需输入对象查询“train”和元数据查询“Germany”。采用向量搜索技术，系统可高效支持百万级图像或数千小时视频的检索。其模块化架构便于集成新模型。支持本地部署，适用于私有或敏感数据集，并已应用于多个真实场景。代码开源，地址：https://gitlab.com/vgg/wise/wise。
原文摘要 · Abstract (English)
In this paper, we present WISE, an open-source audiovisual search engine which integrates a range of multimodal retrieval capabilities into a single, practical tool accessible to users without machine learning expertise. WISE supports natural-language and reverse-image queries at both the scene level (e.g. empty street) and object level (e.g. horse) across images and videos; face-based search for specific individuals; audio retrieval of acoustic events using text (e.g. wood creak) or an audio file; search over automatically transcribed speech; and filtering by user-provided metadata. Rich insights can be obtained by combining queries across modalities -- for example, retrieving German trains from a historical archive by applying the object query "train" and the metadata query "Germany", or searching for a face in a place. By employing vector search techniques, WISE can scale to support efficient retrieval over millions of images or thousands of hours of video. Its modular architecture facilitates the integration of new models. WISE can be deployed locally for private or sensitive collections, and has been applied to various real-world use cases. Our code is open-source and available at https://gitlab.com/vgg/wise/wise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。