Thesis
Bayesian methods for robust and active inverse reinforcement learning
- Abstract:
-
For AI systems to be both helpful and safe, they require a clear specification of human preferences and values, and as these systems grow more autonomous, such specifications must be increasingly robust to avoid harmful unintended consequences. Reward functions offer one formalism for representing human preferences, but handdesigning them is notoriously difficult. Inferring rewards from human behaviour offers an alternative, though any finite set of demonstrations leaves the true reward underdetermined. Bayesian inverse reinforcement learning (IRL) addresses this by quantifying uncertainty over the inferred reward, enabling robust policies and principled identification of where additional human input would be most valuable.
However, existing Bayesian IRL methods remain largely impractical. Approaches that meaningfully capture posterior uncertainty are computationally intractable beyond small discrete environments, and all methods demand numerous expert demonstrations that are costly to obtain in terms of human labour.
This thesis addresses both barriers. To address the computational bottleneck, we introduce Q-centric inference: performing Bayesian inference over Q-values rather than rewards to avoid the computational burden of repeated forward planning, while exploiting the inverse Bellman equation to also recover the implied reward posterior. ValueWalk implements this via Hamiltonian Monte Carlo, while Q-based Variational IRL (QVIRL) uses variational inference to scale to higher-dimensional continuous domains including learning from raw-pixel observations in ATARI games. To address the scarcity of human demonstrations, we develop information-theoretic active IRL methods that strategically select maximally informative demonstration scenarios, providing the first probably-approximately-correct (PAC) guarantees for active IRL with imperfect demonstrations.
Actions
Access Document
- Files:
-
-
(Preview, Dissemination version, pdf, 4.3MB, Terms of use)
-
Authors
Contributors
+ Osborne, M
- Institution:
- University of Oxford
- Division:
- MPLS
- Department:
- Engineering Science
- Role:
- Supervisor
- ORCID:
- 0000-0003-1959-012X
+ Abate, A
- Institution:
- University of Oxford
- Division:
- MPLS
- Department:
- Computer Science
- Role:
- Supervisor
+ Gatsis, K
- Institution:
- Villanova University
- Role:
- Supervisor
+
Engineering and Physical Sciences Research Council
More from this funder
- Funding agency for:
- Bajgar, O
- Grant:
- EP/S024050/1
+
Google DeepMind (United Kingdom)
More from this funder
- Funding agency for:
- Bajgar, O
- Programme:
- Deepmind Scholarship
- DOI:
- Type of award:
- DPhil
- Awarding institution:
- University of Oxford
- Language:
-
English
- Keywords:
- Subjects:
- Deposit date:
-
2026-11-01
- ARK identifier:
Terms of use
- Copyright holder:
- Ondrej Bajgar
- Copyright date:
- 2025
- Notes:
- Walking the values in bayesian inverse reinforcement learning is derived from this thesis.
If you are the owner of this record, you can report an update to it here: Report update to this record