整合多种大模型查询系统，让跨模态数据查询更高效准确。
Beyond Single-Modal Analytics: A Framework for Integrating Heterogeneous LLM-Based Query Systems for Multi-Modal Data
- 构建统一查询引擎，打通不同大模型系统的接口壁垒。
- 在多数数据集上提升3-6倍查询准确率，特定场景达24倍。
- 适合需要跨文本、图像、视频等多模态数据查询的开发者。
随着多模态数据应用增多，语义查询在数据管理系统中愈发重要，是访问和分析多模态数据的关键方式。由于文本、图像、视频等非结构化数据的信息隐藏于语义中，传统SQL查询无法直接获取。近年来，基于大语言模型（LLMs）的语义查询系统被提出，以支持对非结构化数据的自然语言查询。然而，这一领域迅速发展导致生态碎片化：不同系统的API不兼容，且存在专精与通用间的根本权衡。许多系统在单一模态上表现优异，但难以处理多模态数据；而“全功能”系统虽能支持多模态，但在特定模态上性能低于专业系统。本文提出Meta Engine，一种‘查询系统之上的查询系统’，可统一集成异构的专用大模型查询系统。其架构包含五个核心组件：自然语言查询解析器、操作符生成器、查询路由模块、适配器集合及结果聚合器。评估显示，Meta Engine在多数情况下显著优于基线，F1值提升3–6倍，特定数据集最高达约24倍。
原文摘要 · Abstract (English)
With the increasing use of multi-modal data, semantic query has become more and more demanded in data management systems, which is an important way to access and analyze multi-modal data. As unstructured data, most information of multi-modal data (text, image, video, etc.) hides in the semantics, which cannot be accessed by traditional database queries like SQL. Given the power of Large Language Models (LLMs) in understanding semantics and processing natural language, in recent years several LLM-based semantic query systems have been proposed to support semantic querying over unstructured data. However, this rapid growth has produced a fragmented ecosystem. Applications face significant integration challenges due to (1) disparate APIs of different semantic query systems and (2) a fundamental trade-off between specialization and generality. Many semantic query systems are highly specialized, offering state-of-the-art performance within a single modality but struggling with multi-modal data. Conversely, some "all-in-one" systems handle multiple modalities but often exhibit suboptimal performance compared to their specialized counterparts in specific modalities. This paper introduces Meta Engine, a novel ``query system on query systems'', designed to resolve those aforementioned challenges. Meta Engine is a unified semantic query engine that integrates heterogeneous, specialized LLM-based query systems. Its architecture comprises five key components: (1) a Natural Language (NL) Query Parser, (2) an Operator Generator, (3) a Query Router, (4) a set of Adapters, and (5) a Result Aggregator. In the evaluation, Meta Engine consistently outperforms all baselines, yielding 3--6x higher F1 in most cases and up to ~24x on specific datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。