ShapeStacks: Learning vision-based physical intuition for generalised object stacking

Conference item

Physical intuition is pivotal for intelligent agents to perform complex tasks. In this paper we investigate the passive acquisition of an intuitive understanding of physical principles as well as the active utilisation of this intuition in the context of generalised object stacking. To this end, we provide ShapeStacks (Source code & data are available at http://shapestacks.robots.ox.ac.uk): a simulation-based dataset featuring 20,000 stack configurations composed of a variety of elementary geometric primitives richly annotated regarding semantics and structural stability. We train visual classifiers for binary stability prediction on the ShapeStacks data and scrutinise their learned physical intuition. Due to the richness of the training data our approach also generalises favourably to real-world scenarios achieving state-of-the-art stability prediction on a publicly available benchmark of block towers. We then leverage the physical intuition learned by our model to actively construct stable stacks and observe the emergence of an intuitive notion of stackability - an inherent object affordance - induced by the active stacking task. Our approach performs well exceeding the stack height observed during training and even manages to counterbalance initially unstable structures.

Authors: Groth, O; Fuchs, F; Posner, H; Vedaldi, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Springer
• Host title: European Conference on Computer Vision (ECCV 2018)
• Journal: European Conference on Computer Vision (ECCV 2018)
• Publication date: 2018-10-06
• Acceptance date: 2018-07-03
• DOI: 10.1007/978-3-030-01246-5_43
• ISSN: 0302-9743 and 1611-3349
• ISBN: 9783030012458
• Keywords: stability prediction; object stacking; intuitive physics