测试大模型写量化交易代码的能力,发现多数能跑通但算不准收益风险。
Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics
- 用自然语言描述策略,让模型生成可回测的代码
- 最简单任务平均4.08/5轮成功执行,复杂任务误差差10倍以上
- 适合研究AI金融应用或模型推理能力的开发者
我们提出MARKET-BENCH,一个评估大语言模型在基础量化交易任务上的基准。模型需根据自然语言描述和市场假设,生成可执行的回测代码。每项任务对应三种经典策略:对微软(NASDAQ: MSFT)进行定时交易、可口可乐(NASDAQ: KO)与百事(NASDAQ: PEP)配对交易、以及对微软的德尔塔对冲。模型输出需匹配参考实现的盈亏(P&L)、回撤和持仓路径。我们采用多轮评估,区分结构可靠性(能否运行)与数值准确性(回测指标均方误差),失败输出以重复指标作为基线。十三个先进模型中,多数在最简单策略上表现稳定(平均4.08/5轮可执行),但不同模型和任务间误差差异达数个数量级。Gemini 3 Pro与Claude 4.5 Sonnet在简单任务上兼具高可靠性和低误差;GPT-5.2整体表现优异且完全可执行;GPT-5.1 Codex-Max在最简单任务中最佳运行误差最低;Qwen3 Max虽完全可执行,但有时盈亏路径不准。结果表明当前大模型可搭建基础交易框架,但在价格、库存与风险推理上仍不稳健。我们已开源MARKET-BENCH与公开排行榜(https://marketbench.ai)。
原文摘要 · Abstract (English)
We introduce MARKET-BENCH, a benchmark that evaluates large language models (LLMs) on introductory quantitative trading tasks by asking them to construct executable backtesters from natural language strategy descriptions and market assumptions. Each instance specifies one of three canonical strategies: scheduled trading on Microsoft (NASDAQ: MSFT), pairs trading on Coca-Cola (NASDAQ: KO) and Pepsi (NASDAQ: PEP), or delta hedging on MSFT. Models must produce code whose profit and loss (P and L), drawdown, and position paths match a verifiable reference implementation. We assess thirteen state-of-the-art models using a multi-round evaluation that separates structural reliability (whether the backtest runs) from numerical accuracy (mean absolute error of the backtest metrics), assigning failed outputs a duplicated-metrics baseline MAE. While most models reliably execute the simplest strategy (average executable passes of 4.08 out of 5 rounds), errors vary by orders of magnitude across models and tasks. Gemini 3 Pro and Claude 4.5 Sonnet combine strong reliability with low error on simpler strategies. GPT-5.2 achieves strong overall performance with perfect executability. GPT-5.1 Codex-Max achieves the lowest best-run error on the easiest task. Qwen3 Max attains perfect executability yet sometimes produces inaccurate profit and loss paths. These results show that current LLMs can scaffold basic trading infrastructure but still struggle to reason robustly about prices, inventory, and risk. We release MARKET-BENCH and a public leaderboard at https://marketbench.ai.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。