Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation

A new research paper submitted to arXiv by George Panagopoulos critically examines the evaluation methodologies used in heterogeneous treatment effect estimation via machine learning. The study highlights a significant disconnect between academic research, which relies on semi-simulated benchmarks and counterfactual metrics, and industrial applications, which prioritize observable metrics based on real-world outcomes. Through a large-scale empirical study covering various meta-learners and specialized causal models, the author identifies two major gaps. First, counterfactual metrics fail to reliably identify the estimators preferred by observable metrics, even within identical semi-simulated environments. Second, performance rankings derived from semi-simulated benchmarks do not effectively transfer to real-world datasets. The findings indicate that simple meta-learners equipped with strong base models often outperform or match specialized causal models. Consequently, the paper argues that the field must move beyond relying solely on counterfactual metrics and semi-simulated data. Instead, it advocates for incorporating observable metrics and rigorous real-data validation to ensure that methodological progress translates into practical deployment success.