让AI通过多模态案例推理，提升问题解决能力。
A General Retrieval-Augmented Generation Framework for Multimodal Case-Based Reasoning Applications
- 将图像、文本等多模态数据转为文本表示，实现统一检索
- 在数学与西洋双陆棋任务中，生成质量显著优于基线模型
- 适合医疗、法律等需结合图文经验的复杂决策场景
案例推理（CBR）是一种基于经验的问题求解方法，通过调用已解决案例来应对新问题。近年来研究发现，具备检索增强生成（RAG）能力的大语言模型（LLM）可支持CBR流程中的“检索”与“复用”阶段，通过查找相似案例并作为额外上下文输入给LLM。然而，多数研究仅限于纯文本应用；而在现实问题中，案例常包含多模态成分。本文提出MCBR-RAG，一种通用的多模态CBR-RAG框架。该框架将非文本案例组件转换为文本表征，从而实现：1）学习特定应用的潜在表示并用于索引检索；2）通过整合所有案例成分丰富LLM查询上下文。我们在简化版数学题（Math-24）和更复杂的西洋双陆棋（Backgammon）应用上进行了实验，结果表明，相较于未提供上下文信息的基线LLM，MCBR-RAG显著提升了生成质量。
原文摘要 · Abstract (English)
Case-based reasoning (CBR) is an experience-based approach to problem solving, where a repository of solved cases is adapted to solve new cases. Recent research shows that Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) can support the Retrieve and Reuse stages of the CBR pipeline by retrieving similar cases and using them as additional context to an LLM query. Most studies have focused on text-only applications, however, in many real-world problems the components of a case are multimodal. In this paper we present MCBR-RAG, a general RAG framework for multimodal CBR applications. The MCBR-RAG framework converts non-text case components into text-based representations, allowing it to: 1) learn application-specific latent representations that can be indexed for retrieval, and 2) enrich the query provided to the LLM by incorporating all case components for better context. We demonstrate MCBR-RAG's effectiveness through experiments conducted on a simplified Math-24 application and a more complex Backgammon application. Our empirical results show that MCBR-RAG improves generation quality compared to a baseline LLM with no contextual information provided.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。