Thesis icon

Thesis

Bayesian methods for robust and active inverse reinforcement learning

Abstract:
For AI systems to be both helpful and safe, they require a clear specification of human preferences and values, and as these systems grow more autonomous, such specifications must be increasingly robust to avoid harmful unintended consequences. Reward functions offer one formalism for representing human preferences, but handdesigning them is notoriously difficult. Inferring rewards from human behaviour offers an alternative, though any finite set of demonstrations leaves the true reward underdetermined. Bayesian inverse reinforcement learning (IRL) addresses this by quantifying uncertainty over the inferred reward, enabling robust policies and principled identification of where additional human input would be most valuable.

However, existing Bayesian IRL methods remain largely impractical. Approaches that meaningfully capture posterior uncertainty are computationally intractable beyond small discrete environments, and all methods demand numerous expert demonstrations that are costly to obtain in terms of human labour.

This thesis addresses both barriers. To address the computational bottleneck, we introduce Q-centric inference: performing Bayesian inference over Q-values rather than rewards to avoid the computational burden of repeated forward planning, while exploiting the inverse Bellman equation to also recover the implied reward posterior. ValueWalk implements this via Hamiltonian Monte Carlo, while Q-based Variational IRL (QVIRL) uses variational inference to scale to higher-dimensional continuous domains including learning from raw-pixel observations in ATARI games. To address the scarcity of human demonstrations, we develop information-theoretic active IRL methods that strategically select maximally informative demonstration scenarios, providing the first probably-approximately-correct (PAC) guarantees for active IRL with imperfect demonstrations.

Actions

Access Document

Files:

Authors

More by this author
Institution:
University of Oxford
Division:
MPLS
Department:
Engineering Science
Research group:
Machine Learning Research Group
Oxford college:
Lady Margaret Hall
Role:
Author
ORCID:
https://orcid.org/0000-0002-6208-1534

Contributors

Institution:
University of Oxford
Division:
MPLS
Department:
Engineering Science
Role:
Supervisor
ORCID:
0000-0003-1959-012X
Institution:
University of Oxford
Division:
MPLS
Department:
Computer Science
Role:
Supervisor
Institution:
Villanova University
Role:
Supervisor


Engineering and Physical Sciences Research Council More from this funder
Funding agency for:
Bajgar, O
Grant:
EP/S024050/1
Google DeepMind (United Kingdom) More from this funder
Funding agency for:
Bajgar, O
Programme:
Deepmind Scholarship


DOI:
Type of award:
DPhil
Awarding institution:
University of Oxford

Terms of use


Views and Downloads

Views and downloads will return soon






If you are the owner of this record, you can report an update to it here: Report update to this record

TO TOP