用分步流程提升大模型处理长文本数值任务的准确率与效率
An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks
- 将复杂任务拆解为判断、提取、编码计算和结论四步
- 在两个基准上准确率提升，API调用成本显著降低
- 适合需要高效处理长文本数值计算的场景
大型语言模型（LLMs）在处理长文本方面表现出色，传统检索任务几乎完美。但在涉及数值计算的长上下文任务中，其性能显著下降。现有方法依赖思维链提示，虽可提高准确率，但需大量输出token，成本高且耗时。为此，我们提出一种工作流，将数值相关长上下文任务分解为四个低级子任务：判断、提取、代码计算和结论。前两个任务较简单，可用小型模型高效处理长上下文。涉及数值计算时，使用LLM生成代码，避免模型不擅长计算的缺陷。在两个数值相关长上下文基准上的实验表明，该方法不仅提升了准确率，还显著降低了API调用成本。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long texts and have almost perfect performance in traditional retrieval tasks. However, their performance significantly degrades when it comes to numerical calculations in the long-context. Numeric-involved long-context tasks typically cannot be addressed by current LLMs in normal settings due to their inherent limitations in simultaneously handling complex and massive information. Some CoT like prompting methods can improve accuracy but demands massive output tokens, which is costly and slow. To address this issue, we propose a workflow, which decompose a numeric-involved long-context task into 4 low-level subtasks: judging, extracting and processing with code and conclusion. The former 2 subtasks is relatively simple, which allows us to use smaller models for efficiently processing long context. When numerical calculations are required, we use code generated by LLMs to avoid the disadvantage of LLM not being good at calculations. The results in 2 numeric-involved long-context benchmarks demonstrate our workflow can not only improve accuracy, but also significantly reduce the cost of API calls.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。