Learning affordances in object-centric generative models

Conference item : Poster

Given visual observations of a reaching task together with a stick-like tool, we propose a novel approach that learns to exploit task-relevant object affordances by combining generative modelling with a task-based performance predictor. The embedding learned by the generative model captures the factors of variation in object geometry, e.g. length, width, and configuration. The performance predictor identifies sub-manifolds correlated with task success in a weakly supervised manner. Using a 3D simulation environment, we demonstrate that traversing the latent space in this task-driven way results in appropriate tool geometries for the task at hand. Our results suggest that affordances are encoded along smooth trajectories in the learned latent space. Given only high-level performance criteria (such as task success), accessing these emergent affordances via gradient descent enables the agent to manipulate learned object geometries in a targeted and deliberate way.

Authors: Wu, Y; Kasewa, S; Groth, O; Salter, S; Sun, L

• Publication status: Published
• Peer review status: Reviewed (other)
• Publisher: International Conference on Machine Learning
• Publication date: 2020-07-17
• Acceptance date: 2020-06-01
• Event title: Workshop on Object-Oriented Learning at ICML 2020
• Event series: International Conference on Machine Learning
• Event location: Online
• Event website: https://icml.cc/Conferences/2020
• Event start date: 2020-07-12
• Event end date: 2020-07-18
• Language: English
• Keywords: FFR
• Subtype: Poster