让大模型通过概念引导解决复杂工程问题,准确率提升超三成。
Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Models
- 用概念上下文学习与概念链增强模型推理能力
- 相比传统方法,答案正确率提升30.6%至29.88%
- 适合需精准推理的工程建模与科学任务
科学与工程问题属于需要特定概念信息(CI)的复杂概念问题,如数学逻辑知识、流程信息或工程规范。大语言模型(LLMs)因在辅助解决问题方面的潜力,被视为解决此类问题的有力工具。但通用训练的原始LLMs缺乏必要概念信息。本文探索浅层定制方法(SCMs),提出两种新算法:概念上下文学习(C-ICL)与概念链(CoC),以增强LLMs的概念能力,使其能解决复杂概念问题。研究聚焦于基于数据建模指南生成行业专属数据模型的任务。在多种规模的OpenAI LLM上,使用语法和语义正确性、耗时与成本四项指标进行评估。结果表明,新方法优于主流的ICL和CoT。相比CoT,C-ICL与CoC分别将回答正确率提升30.6%和29.88%。定性分析显示,新方法激活了模型此前未显现的涌现能力,使推理过程更透明,减少幻觉与提示抄写现象。
原文摘要 · Abstract (English)
Science and engineering problems fall in the category of complex conceptual problems that require specific conceptual information (CI) like math/logic -related know-how, process information, or engineering guidelines to solve them. Large Language Models (LLMs) are promising agents to solve such complex conceptual problems due to their implications in advancing engineering and science tasks like assisted problem-solving. But vanilla LLMs, trained on open-world data, lack the necessary CI. In this work, we specifically explore shallow customization methods (SCMs) of LLMs for solving complex conceptual problems. We propose two novel SCM algorithms for LLM, to augment LLMs with CI and enable LLMs to solve complex conceptual problems: Conceptual In-Context Learning (C-ICL) and Chain of Concepts (CoC). The problem tackled in this paper is generation of proprietary data models in the engineering/industry domain based on conceptual information in data modelling guidelines. We evaluate our algorithms on varied sizes of the OpenAI LLMs against four evaluation metrics related to syntactic and semantic correctness, time and cost incurred. The proposed algorithms perform better than currently popular LLM SCMs like In-context Learning (ICL) and Chain of Thoughts (CoT). It was observed that as compared to CoT, response correctness increased by 30.6% and 29.88% for the new SCMs C-ICL and CoC respectively. Qualitative analysis suggests that the proposed new SCMs activate emergent capabilities in LLMs, previously unobserved in the existing SCMs. They make problem-solving processes more transparent and reduce hallucinations and the tendency of model responses to copy examples from prompts (parroting).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。