During training, model-free reinforcement learning (RL) systems can explore actions that lead to harmful or costly consequences. Having a human “in the loop” and ready to intervene at all times can prevent these mistakes, but is prohibitively expensive for current algorithms. We explore how human oversight can be combined with a supervised learning system to prevent catastrophic events during training. We demonstrate this scheme on Atari games, with a Deep RL agent being overseen by a human f...