Towards representation learning for treatment effect estimation with high-dimensional covariates

Thesis

Treatment effect estimation is typically difficult in the presence of high-dimensional covariates. In this thesis, we propose to use a representation, that is, the mapping of covariates to a lower-dimensional manifold, as an adjustment set in estimators of treatment effects.

In an introductory chapter, we review treatment effect estimation as well as challenges with high-dimensional features, both in statistics or machine learning generally and in treatment effect estimation in particular. The thesis’s contributions are then outlined.

As a first contribution, we extend the popular propensity score matching method to the use of multivariate representations in matching. This can be done by noticing that neural networks naturally satisfy the compositional nature of balancing scores from Rosenbaum and Rubin [1983], which are valid adjustment sets. As a result, intermediary hidden layers of a neural network modelling the propensity score can be used as adjustment sets. We also bound the original covariate imbalance using the representation imbalance in different settings. This method requires that the neural network model is correctly specified.

We address this in the second contribution, where we upper-bound the bias induced by adjusting for a representation instead of original covariates using a loss which builds on former work in density ratio regression and is differentiable. As a result, a measure of misspecification of the representation, which is added to the representation imbalance in the former bound, can directly be targeted. We also extend the analysis from matching to a more general weighting framework.

In a third contribution, we address the problem of poor overlap which is typical for high-dimensional covariates. We quantify the degree of overlap of a representation using a specific “overlap divergence” measure, and justify doing so by connecting it to the variance of estimators adjusting for the representation. We establish that outcome information is required to find a representation improving overlap, which stands in contrast to the two previous contributions. In a simplified setting with Gaussian covariates and generalised linear models for the outcome model and the propensity score model, we further strengthen the result: we describe a class of representations without confounding bias where the more predictive of the outcome the representation, the better its overlap.

We conclude with a summary of the contributions, key findings and limitations in all methods.

Authors: Clivio, O

• DOI: 10.5287/ora-vjeeyo7pd
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: treatment effect estimation; causal inference; representation learning; high dimensions
• Subjects: Statistics; Machine Learning