开源网页工具提升大模型对话理解，支持自定义文档检索与话语行为分析。
An Open-Source Web-Based Tool for Evaluating Open-Source Large Language Models Leveraging Information Retrieval from Custom Documents
- 构建可上传文档的网页平台，让大模型基于用户话语行为检索信息回应。
- 实验验证：大模型在包含话语行为时对话对齐度更高，小模型则困惑度上升。
- 适合对话系统研究者、语言模型评估者使用，尤其关注语用层面优化。
本文提出首个开源的基于网页的大语言模型评估工具，用于分析用户在对话中的话语行为对模型响应的影响。该工具支持上传特定文档，供对话模型进行信息检索，并通过提取语言特征生成上下文嵌入，从而可视化用户的沟通意图。尽管现有模型已具备一定上下文理解能力，但缺乏深层语用特征。为此，本工具将被忽略的语用特征注入编码器-解码器流程，以增强理解。实验在5个开源大模型上，使用2个知识文件和2个用户查询，通过10项标准指标进行评估。结果显示，大模型在引入话语行为后对齐度显著提升；而小模型则出现困惑度升高、表现不稳，表明其处理含话语行为查询存在困难。分析表明，话语行为有助于深化对话，但也需针对不同模型优化计算开销与响应延迟。
原文摘要 · Abstract (English)
In our work, we present the first-of-its-kind open-source web-based tool which is able to demonstrate the impacts of a user's speech act during discourse with conversational agents, which leverages open-source large language models. With this software resource, it is possible for researchers and experts to evaluate the performance of various dialogues, visualize the user's communicative intents, and utilise uploaded specific documents for the chat agent to use for its information retrieval to respond to the user query. The context gathered by these models is obtained from a set of linguistic features extracted, which forms the context embeddings of the models. Regardless of these models showing good context understanding based on these features, there still remains a gap in including deeper pragmatic features to improve the model's comprehension of the query, hence the efforts to develop this web resource, which is able to extract and then inject this overlooked feature in the encoder-decoder pipeline of the conversational agent. To demonstrate the effect and impact of the resource, we carried out an experiment which evaluated the system using 2 knowledge files for information retrieval, with two user queries each, across 5 open-source large language models using 10 standard metrics. Our results showed that larger open-source models, demonstrated an improved alignment when the user speech act was included with their query. The smaller models in contrast showed an increased perplexity and mixed performance, which explicitly indicated struggles in processing queries that explicitly included speech acts. The results from the analysis using the developed web resource highlight the potential of speech acts towards enhancing conversational depths while underscoring the need for model-specific optimizations to address increased computational costs and response times.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。