评测大模型生成可执行量化交易策略的能力，发现核心瓶颈在逻辑与业务对齐。
QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
- 构建400个真实场景的交易策略任务，评估模型从描述生成可运行代码的能力。
- 多阶段验证：语法正确、回测成功、产生交易、符合需求，失败主要在逻辑和API使用。
- 揭示大模型在金融领域代码生成中更缺的是业务理解而非编程能力，适合量化研究者参考。
大型语言模型在通用编程任务上表现优异，但在生成可执行量化交易策略方面仍缺乏深入研究。与常规代码基准不同，交易策略生成需同时掌握金融领域知识、专用API使用以及生成能实际执行的代码。本文提出QuantCode-Bench，用于系统评估现代大模型基于英文描述，在Backtrader框架下生成交易策略的能力。该基准包含400个来自Reddit、TradingView、StackExchange、GitHub及合成来源的不同时难任务。评估采用多阶段流水线：检查语法正确性、回测能否成功执行、是否产生实际交易，以及通过大模型判断与任务描述的语义一致性。对比了先进模型在单次生成和代理式多轮交互两种设置下的表现。分析各阶段失败模式发现，当前模型的主要限制并非语法错误，而是交易逻辑操作不当、API使用错误以及任务语义偏离。结果表明，交易策略生成属于独特的领域特定代码生成任务，成功不仅依赖技术正确性，还需自然语言描述、金融逻辑与策略在数据上的可观测行为三者对齐。
原文摘要 · Abstract (English)
Large language models have demonstrated strong performance on general-purpose programming tasks, yet their ability to generate executable algorithmic trading strategies remains underexplored. Unlike standard code benchmarks, trading-strategy generation requires simultaneous mastery of domain-specific financial logic, knowledge of a specialized API, and the ability to produce code that is not only syntactically correct but also leads to actual trades on historical data. In this work, we present QuantCode-Bench, a benchmark for the systematic evaluation of modern LLMs in generating strategies for the Backtrader framework from textual descriptions in English. The benchmark contains 400 tasks of varying difficulty collected from Reddit, TradingView, StackExchange, GitHub, and synthetic sources. Evaluation is conducted through a multi-stage pipeline that checks syntactic correctness, successful backtest execution, the presence of trades, and semantic alignment with the task description using an LLM judge. We compare state-of-the-art models in two settings: single-turn, where the strategy must be generated correctly on the first attempt, and agentic multi-turn, where the model receives iterative feedback and may repair its errors. We analyze the failure modes across different stages of the pipeline and show that the main limitations of current models are not related to syntax, but rather to the correct operationalization of trading logic, proper API usage, and adherence to task semantics. These findings suggest that trading strategy generation constitutes a distinct class of domain-specific code generation tasks in which success requires not only technical correctness, but also alignment between natural-language descriptions, financial logic, and the observable behavior of the strategy on data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。