构建多模态合成数据检测基准，评估大模型识别真假内容能力
LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models
- 设计跨图像、视频、3D、文本、音频的多模态检测任务
- 包含18,000个问题，覆盖26个子类别，涵盖粗粒度到细粒度判断
- 适用于测试大模型感知与推理能力，助力可信AI发展
随着人工智能生成内容的快速发展，未来互联网可能充斥合成数据，真实与可信多模态内容的鉴别变得愈发困难。合成数据检测因此受到广泛关注，大型多模态模型（LMMs）在此任务中的表现尤其引人注目。LMMs能为真实性判断提供自然语言解释，提升检测可解释性，同时该任务也能有效检验LMMs的感知、知识与推理能力。为此，我们提出LOKI——一个新型基准，用于评估LMMs在多种模态下检测合成数据的能力。LOKI涵盖视频、图像、3D、文本和音频模态，包含18,000个精心设计的问题，分布在26个子类别中，具有明确难度分级。基准包括粗粒度判断、多项选择题，以及细粒度异常定位与解释任务，支持对LMMs能力的全面分析。我们在LOKI上评估了22个开源与6个闭源LMM模型，揭示其作为合成数据检测器的潜力，同时也暴露了当前大模型能力发展的局限。更多LOKI信息请访问 https://opendatalab.github.io/LOKI/
原文摘要 · Abstract (English)
With the rapid development of AI-generated content, the future internet may be inundated with synthetic data, making the discrimination of authentic and credible multimodal data increasingly challenging. Synthetic data detection has thus garnered widespread attention, and the performance of large multimodal models (LMMs) in this task has attracted significant interest. LMMs can provide natural language explanations for their authenticity judgments, enhancing the explainability of synthetic content detection. Simultaneously, the task of distinguishing between real and synthetic data effectively tests the perception, knowledge, and reasoning capabilities of LMMs. In response, we introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to detect synthetic data across multiple modalities. LOKI encompasses video, image, 3D, text, and audio modalities, comprising 18K carefully curated questions across 26 subcategories with clear difficulty levels. The benchmark includes coarse-grained judgment and multiple-choice questions, as well as fine-grained anomaly selection and explanation tasks, allowing for a comprehensive analysis of LMMs. We evaluated 22 open-source LMMs and 6 closed-source models on LOKI, highlighting their potential as synthetic data detectors and also revealing some limitations in the development of LMM capabilities. More information about LOKI can be found at https://opendatalab.github.io/LOKI/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。