Planning for risk-aversion and expected value in MDPs

Conference item

Planning in Markov decision processes (MDPs) typically optimises the expected cost. However, optimising the expectation does not consider the risk that for any given run of the MDP, the total cost received may be unacceptably high. An alternative approach is to find a policy which optimises a risk-averse objective such as conditional value at risk (CVaR). However, optimising the CVaR alone may result in poor performance in expectation. In this work, we begin by showing that there can be multiple policies which obtain the optimal CVaR. This motivates us to propose a lexicographic approach which minimises the expected cost subject to the constraint that the CVaR of the total cost is optimal. We present an algorithm for this problem and evaluate our approach on four domains. Our results demonstrate that our lexicographic approach improves the expected cost compared to the state of the art algorithm, while achieving the optimal CVaR.

Authors: Rigter, M; Duckworth, P; Lacerda, B; Hawes, N

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Association for the Advancement of Artificial Intelligence
• Host title: Proceedings of the 32nd International Conference on Automated Planning and Scheduling (ICAPS 2022)
• Volume: 32
• Issue: 1
• Pages: 307-315
• Publication date: 2022-06-13
• Event title: 32nd International Conference on Automated Planning and Scheduling (ICAPS 2022)
• Event location: Singapore
• Event website: https://icaps22.icaps-conference.org/
• Event start date: 2022-06-13
• Event end date: 2022-06-24
• DOI: 10.1609/icaps.v32i1.19814
• EISSN: 2334-0843
• ISSN: 2334-0835
• ISBN: 9781577358749
• Language: English
• Keywords: FFR