GENESIS: generative scene inference and sampling of object-centric latent representations

Conference item

Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of rendered 3D scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.

Authors: Engelcke, M; Kosiorek, AR; Parker Jones, O; Posner, H

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: OpenReview
• Journal: Proceedings of the ICLR 2020
• Publication date: 2020-03-11
• Acceptance date: 2019-12-19
• Event title: International Conference on Learning Representations 2020 (ICLR)
• Event location: Online
• Event website: https://iclr.cc/Conferences/2020
• Event start date: 2020-04-26
• Event end date: 2020-05-01
• Language: English
• Keywords: generation; FFR; generative models; reinforcement learning; unsupervised; robotics; semi supervised learning; variational inference