用大模型生成+机器学习优化,让特征转换更稳定准确
LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformation
- 大模型负责生成语法正确的特征,机器学习通过梯度搜索提升稳定性
- 下游任务性能提升5%,错误案例减少近一半
- 适合需要高可靠性特征工程的科研与工业场景
特征转换通过从原始数据中衍生新特征来增强数据表示。生成式AI在此任务中具有潜力,但面临生成不稳定(输出不一致)和生成无效(序列出错)两大挑战。现有方法中,传统机器学习有效性低,大语言模型(LLM)则缺乏稳定性。我们发现,LLM能保证语法正确性,而机器学习的梯度搜索可稳定性能。为此,提出一种融合框架:首先生成高质量示例(基于教师模型知识);其次在隐空间中嵌入并搜索潜在优解;再次让学生模型从教师模型中提炼知识;最后通过联合概率实现有效且稳定的生成。在多个数据集上的实验表明,该策略使下游性能提升5%,错误案例减少近半,且具高效性与鲁棒性。此外还发现,大模型具备理解原始数据的能力。
原文摘要 · Abstract (English)
Feature transformation enhances data representation by deriving new features from the original data. Generative AI offers potential for this task, but faces challenges in stable generation (consistent outputs) and valid generation (error-free sequences). Existing methods--traditional MLs' low validity and LLMs' instability--fail to resolve both. We find that LLMs ensure valid syntax, while ML's gradient-steered search stabilizes performance. To bridge this gap, we propose a teaming framework combining LLMs' symbolic generation with ML's gradient optimization. This framework includes four steps: (1) golden examples generation, aiming to prepare high-quality samples with the ground knowledge of the teacher LLM; (2) feature transformation sequence embedding and search, intending to uncover potentially superior embeddings within the latent space; (3) student LLM feature transformation, aiming to distill knowledge from the teacher LLM; (4) LLM-ML decoder teaming, dedicating to combine ML and the student LLM probabilities for valid and stable generation. The experiments on various datasets show that the teaming policy can achieve 5\% improvement in downstream performance while reducing nearly half of the error cases. The results also demonstrate the efficiency and robustness of the teaming policy. Additionally, we also have exciting findings on LLMs' capacity to understand the original data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。