让大模型通过上下文学习搞定机器学习任务，支持上千示例精准推理。
MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining
- 用因果模型生成海量机器学习任务，持续预训练增强上下文学习能力
- 在1024个示例下准确率持续上升，达随机森林水平，比基线高15%
- 保留通用对话能力，适合需复杂推理的跨领域应用
大型语言模型具备广泛世界知识和强通用推理能力，但在标准机器学习任务上难以仅通过上下文学习（ICL）从大量示例中学习，即不使用梯度下降。我们提出MachineLearningLM，一种可移植的持续预训练框架，使通用大模型具备稳健的上下文学习机器学习能力，同时保持其通用知识与推理能力，适用于更广泛的对话流程。预训练过程从数百万个结构因果模型（SCMs）中合成机器学习任务，覆盖最多1,024个示例。我们以随机森林教师开始，将基于树的决策策略蒸馏至大模型中，提升数值建模鲁棒性。所有任务采用令牌高效提示序列化，使每个上下文窗口内示例数量增加3倍至6倍，通过批量推理实现高达50倍的摊销吞吐量提升。尽管设置简单（Qwen-2.5-7B-Instruct + LoRA rank 8），MachineLearningLM在金融、物理、生物和医疗等领域的跨分布表格分类任务上，平均优于强基线（如GPT-5-mini）约15%。其表现出显著的多示例扩展规律：准确率随上下文示例数从8增至1,024单调上升。无需任务特定训练，即可在数百个示例下达到随机森林级别的准确率。通用聊天能力得以保留：在MMLU上取得75.4%得分。
原文摘要 · Abstract (English)
Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。