用大模型嵌入代替传统回归，实现跨领域的高效黑盒优化
Language Model Embeddings Can Be Sufficient for Bayesian Optimization
- 将输入转为字符串，利用LLM嵌入做上下文回归
- 在合成、组合及超参优化中性能媲美谷歌Vizier
- 适合需要灵活搜索空间的实验设计场景
贝叶斯优化广泛用于实验设计和黑盒优化，以提升搜索效率。但现有方法多依赖回归模型，受限于固定搜索空间和结构化表格特征。本文探索使用大型语言模型（LLM）对字符串输入进行嵌入，实现上下文中的回归任务。结果表明，将输入表示为字符串可实现跨多样领域的通用回归，涵盖合成、组合与超参数优化。该方法性能与最先进的基于高斯过程的方法（如Google Vizier）相当，并展现出更广泛、更灵活的应用潜力。
原文摘要 · Abstract (English)
Bayesian Optimization is ubiquitous in experimental design and black-box optimization for improving search efficiency. However, most existing approaches rely on regression models which are limited to fixed search spaces and structured, tabular input features. This paper explores the use of LLM embeddings over string inputs for in-context regression in Bayesian Optimization. Our results show that representing inputs as strings enables general-purpose regression across diverse domains, including synthetic, combinatorial, and hyperparameter optimization. Furthermore, our approach achieves optimization performance comparable to state-of-the-art Gaussian Process-based methods such as Google Vizier, and demonstrates potential for broader and more flexible applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。