LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers

Researchers have introduced LLM-FE, a novel framework designed to enhance automated feature engineering for tabular learning tasks by integrating Large Language Models (LLMs) with evolutionary search algorithms. Traditional methods often rely on fixed, manually designed search spaces and neglect domain knowledge, while existing LLM-based approaches fail to leverage insights from prior experiments or establish meaningful reasoning between feature generation and performance. LLM-FE addresses these limitations by formulating feature engineering as a program search problem. In this process, LLMs iteratively propose new feature transformation programs, guided by data-driven feedback and their inherent reasoning capabilities. This approach allows the system to effectively incorporate domain knowledge and optimize feature selection based on validation scores and historical discovery insights. Experimental results demonstrate that LLM-FE consistently outperforms state-of-the-art baselines, significantly improving predictive model performance across diverse classification and regression benchmarks. The study highlights the potential of combining generative AI with evolutionary optimization to automate complex data preprocessing tasks. The source code for the framework has been made publicly available to facilitate further research and application in machine learning workflows.