An alternative to variance: Gini deviation for risk-averse policy gradient

Conference item

Restricting the variance of a policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.

Authors: Luo, Y; Liu, G; Poupart, P; Pan, Y

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Curran Associates
• Host title: Advances in Neural Information Processing Systems 36
• Volume: 36
• Pages: 60922-60946
• Publication date: 2024-07-01
• Event title: 37th Conference on Neural Information Processing Systems (NeurIPS 2023)
• Event location: New Orleans, Louisiana, USA
• Event website: https://neurips.cc/Conferences/2023
• Event start date: 2023-12-10
• Event end date: 2023-12-16
• DOI: 10.52202/075280-2662
• ISSN: 1049-5258
• EISBN: 9781713899921
• Language: English