提升大模型上下文预测能力的四大策略,兼顾准确性与效率
Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs
- 提出诊断、增效、提准三维度四策略框架
- 准确率提升25-50%,显著缩小执行差距
- 支持小大模型动态调度,降低推理成本
真实世界预测需融合历史数据与文本形式的上下文信息。尽管大语言模型(LLMs)在上下文辅助预测中展现潜力,但仍面临三大挑战:缺乏故障诊断工具、性能远低于预期、计算成本过高。本文提出一个统一框架,包含四个跨维度策略,覆盖模型诊断、准确性与效率。通过在从开源小模型到前沿模型(Gemini、GPT、Claude)的广泛评估中验证,发现:诊断策略揭示了‘执行差距’——模型能正确解释上下文影响却未能应用;准确性策略实现25%-50%的性能提升;效率策略表明,小大模型自适应路由可平均接近大模型精度,同时显著降低推理开销。这些策略可按部署需求灵活组合,为实际应用提供完整工具包。代码已开源:https://github.com/ashok-arjun/beyond-naive-prompting。
原文摘要 · Abstract (English)
Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the "Execution Gap" where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at https://github.com/ashok-arjun/beyond-naive-prompting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。