Revisiting the global workspace orchestrating the hierarchical organization of the human brain

Journal article

Joint-Embedding Predictive Architectures (JEPAs) have recently emerged as a novel and powerful technique for self-supervised representation learning. They aim to learn an energy-based model by predicting the latent representation of a target signal $y$ from a context signal $x$. JEPAs bypass the need for data augmentation and negative samples, which are typically required by contrastive learning, while avoiding the overfitting issues associated with generative-based pretraining. In this paper, we show that graph-level representations can be effectively modeled using this paradigm and propose Graph-JEPA, the first JEPA for the graph domain. In particular, we employ masked modeling to learn embeddings for different subgraphs of the input graph. To endow the representations with the implicit hierarchy that is often present in graph-level concepts, we devise an alternative training objective that consists of predicting the coordinates of the encoded subgraphs on the unit hyperbola in the 2D plane. Extensive validation shows that Graph-JEPA can learn representations that are expressive and competitive in both graph classification and regression problems.Comment: Preprint. Under Revie

Authors: Deco, G; Vidaurre, D; Kringelbach, ML

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Nature Research
• Journal: Nature Human Behaviour
• Volume: 5
• Issue: 4
• Pages: 497-511
• Publication date: 2021-01-04
• DOI: 10.1038/s41562-020-01003-6
• EISSN: 2397-3374
• ISSN: 2397-3374
• Language: English
• Keywords: Neuroscience; Cognitive science; Workspace; Artificial intelligence; Psychology; Data science; Computer science; Human–computer interaction