Learning optimal and sample-efficient decision policies with guarantees

Thesis

The paradigm of decision-making has been revolutionised by reinforcement learning and deep learning due to their ability to learn intelligent agents in unknown environments. Although this has led to significant progress in domains such as robotics, healthcare, and finance, the use of reinforcement learning in practice is challenging, particularly when learning decision policies in high-stakes applications that may require guarantees. Traditional reinforcement learning algorithms rely on online interactions with the environment to gather data for policy learning and are often data-inefficient. This is problematic in scenarios where online interactions are costly, dangerous, or infeasible. In this thesis, we work in the setting of offline reinforcement learning, which assumes access to fixed sets of pre-collected data, and develop algorithms for learning effective policies with provable guarantees. However, learning from offline datasets is hindered by the presence of hidden confounders, that is, variables that influence the actions and the outcome but are not observed in the offline dataset. Such confounders can cause spurious correlations in the dataset and can mislead the agent into taking suboptimal or adversarial actions. Firstly, we address the problem of learning from offline datasets in the presence of hidden confounders within the context of reinforcement learning while ensuring sample efficiency and optimality guarantees. To achieve this, we work with instrumental variables (IVs) to identify the causal effect, which we show is an instance of a conditional moment restrictions (CMR) problem. Inspired by double/debiased machine learning, we derive a novel Neumann orthogonal score function and design a cross-fitting regime to obtain a sampleefficient algorithm for solving CMR problems with convergence and optimality guarantees, which outperforms state-of-the-art on IV regression and proximal causal learning problems. Secondly, we relax the conditions on the hidden confounders in the setting of (offline) imitation learning, and adapt our CMR estimator to derive an algorithm that can learn effective imitator policies with convergence rate guarantees, which outperforms the state of the art. Finally, we consider the problem of learning high-level objectives expressed in linear temporal logic (LTL) and develop a provably optimal learning algorithm that improves sample efficiency over existing methods. The derived methods have been implemented and are made available in open source. Through evaluation on reinforcement learning benchmarks and synthetic and semi-synthetic datasets, we demonstrate the usefulness of the methods developed in this thesis in real-world decision making.

Authors: Shao, D

• DOI: 10.5287/ora-yqpjbpadm
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English