arXiv:2505.18831cs.IR2025-05被引 3

让大模型更懂复杂搜索,能推理还能处理图文。

Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning

  • 用自然语言替代代码表示搜索计划,节省计算资源。
  • 结合强化学习与反馈优化,推理搜索准确率提升54.54%。
  • 支持图文输入输出,适合金融新闻等复杂场景使用。

现有大型语言模型驱动的搜索代理通常依赖提示工程将用户查询拆解为搜索计划,难以应对需要推理的复杂场景。同时,其因采用基于Python的计划表示导致令牌消耗过高,且对多媒体元素的输入处理与输出生成整合不足。为此,我们提出SearchExpert训练方法,以增强大模型在复杂查询下的多模态搜索能力。首先,将搜索计划重构为高效的自然语言表示,降低令牌开销;其次,设计监督微调搜索(SFTS)方法,并构建自动化数据集生成流程;第三,提出基于搜索反馈的强化学习(RLSF),以模型规划的搜索结果作为奖励信号;第四,引入多模态理解与生成代理,使微调后的模型可在推理时处理视觉输入并生成视觉输出。最后,建立自动化基准构建流程与人工评估框架。所提出的SearchExpertBench-25包含200道涵盖金融与国际新闻场景的多选题,需推理能力完成搜索。实验表明,SearchExpert在FinSearchBench-24上比商用方案Perplexity Pro提升36.60%,在SearchExpertBench-25上提升54.54%。人工评估也证实其响应可读性更优。

原文摘要 · Abstract (English)

Existing large language models (LLMs) driven search agents typically rely on prompt engineering to decouple the user queries into search plans, limiting their effectiveness in complex scenarios requiring reasoning. Furthermore, they suffer from excessive token consumption due to Python-based search plan representations and inadequate integration of multimedia elements for both input processing and response generation. To address these challenges, we introduce SearchExpert, a training method for LLMs to improve their multimedia search capabilities in response to complex search queries. Firstly, we reformulate the search plan in an efficient natural language representation to reduce token consumption. Then, we propose the supervised fine-tuning for searching (SFTS) to fine-tune LLM to adapt to these representations, together with an automated dataset construction pipeline. Secondly, to improve reasoning-intensive search capabilities, we propose the reinforcement learning from search feedback (RLSF) that takes the search results planned by LLM as the reward signals. Thirdly, we propose a multimedia understanding and generation agent that enables the fine-tuned LLM to process visual input and produce visual output during inference. Finally, we establish an automated benchmark construction pipeline and a human evaluation framework. Our resulting benchmark, SearchExpertBench-25, comprises 200 multiple-choice questions spanning financial and international news scenarios that require reasoning in searching. Experiments demonstrate that SearchExpert outperforms the commercial LLM search method (Perplexity Pro) by 36.60% on the existing FinSearchBench-24 benchmark and 54.54% on our proposed SearchExpertBench-25. Human evaluations further confirm the superior readability.

多模态搜索大模型推理强化学习信息检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。