Inverse reinforcement learning from failure

Conference item

Inverse reinforcement learning (IRL) allows autonomous agents to learn to solve complex tasks from successful demonstrations. However, in many settings, e.g., when a human learns the task by trial and error, failed demonstrations are also readily available. In addition, in some tasks, purposely generating failed demonstrations may be easier than generating successful ones. Since existing IRL methods cannot make use of failed demonstrations, in this paper we propose inverse reinforcement learning from failure (IRLF) which exploits both successful and failed demonstrations. Starting from the state-of-the-art maximum causal entropy IRL method, we propose a new constrained optimisation formulation that accommodates both types of demonstrations while remaining convex. We then derive update rules for learning reward functions and policies. Experiments on both simulated and real-robot data demonstrate that IRLF converges faster and generalises better than maximum causal entropy IRL, especially when few successful demonstrations are available.

Authors: Shiarlis, K; Messias, J; Whiteson, S

• Publication status: Accepted
• Peer review status: Peer reviewed
• Publisher: International Foundation for Autonomous Agents and Multiagent Systems
• Host title: AAMAS '16 Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems
• Pages: 1060-1068
• Publication date: 2016-01-01
• Acceptance date: 2016-01-15
• ISBN: 9781450342391
• Keywords: machine learning; robotics; inverse reinforcement learning; social navigation; learning from demonstration