Robust anytime learning of Markov decision processes

Conference item

Markov decision processes (MDPs) are formal models commonly used in sequential decision-making. MDPs capture the stochasticity that may arise, for instance, from imprecise actuators via probabilities in the transition function. However, in data-driven applications, deriving precise probabilities from (limited) data introduces statistical errors that may lead to unexpected or undesirable outcomes. Uncertain MDPs (uMDPs) do not require precise probabilities but instead use so-called uncertainty sets in the transitions, accounting for such limited data. Tools from the formal verification community efficiently compute robust policies that provably adhere to formal specifications, like safety constraints, under the worst-case instance in the uncertainty set. We continuously learn the transition probabilities of an MDP in a robust anytime-learning approach that combines a dedicated Bayesian inference scheme with the computation of robust policies. In particular, our method (1) approximates probabilities as intervals, (2) adapts to new data that may be inconsistent with an intermediate model, and (3) may be stopped at any time to compute a robust policy on the uMDP that faithfully captures the data so far. Furthermore, our method is capable of adapting to changes in the environment. We show the effectiveness of our approach and compare it to robust policies computed on uMDPs learned by the UCRL2 reinforce

Authors: Suilen, M; Simão, TD; Jansen, N; Parker, D

• Publication status: Accepted
• Peer review status: Peer reviewed
• Publisher: Curran Associates
• Host title: Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS 2022)
• Volume: 35
• Publication date: 2023-04-01
• Acceptance date: 2022-09-14
• Event title: 36th Annual Conference on Neural Information Processing Systems (NeurIPS 2022)
• Event location: New Orleans, LA, USA
• Event website: https://nips.cc/
• Event start date: 2022-11-28
• Event end date: 2022-12-09
• ISBN: 9781713871088
• Language: English
• Keywords: FFR