Separation power of equivariant neural networks

Conference item : Poster

The separation power of a machine learning model refers to its ability to distinguish between different inputs and is often used as a proxy for its expressivity. Indeed, knowing the separation power of a family of models is a necessary condition to obtain fine-grained universality results. In this paper, we analyze the separation power of equivariant neural networks, such as convolutional and permutation-invariant networks. We first present a complete characterization of inputs indistinguishable by models derived by a given architecture. From this results, we derive how separability is influenced by hyperparameters and architectural choices—such as activation functions, depth, hidden layer width, and representation types. Notably, all non-polynomial activations, including ReLU and sigmoid, are equivalent in expressivity and reach maximum separation power. Depth improves separation power up to a threshold, after which further increases have no effect. Adding invariant features to hidden representations does not impact separation power. Finally, block decomposition of hidden representations affects separability, with minimal components forming a hierarchy in separation power that provides a straightforward method for comparing the separation power of models.

Authors: Pacini, M; Dong, X; Lepri, B; Santin, G

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: OpenReview
• Host title: Proceedings of the 13th International Conference on Learning Representations (ICLR 2025)
• Article number: 10616
• Publication date: 2025-01-22
• Acceptance date: 2025-01-22
• Event title: 13th International Conference on Learning Representations (ICLR 2025)
• Event location: Singapore
• Event website: https://iclr.cc/Conferences/2025
• Event start date: 2025-04-24
• Event end date: 2025-04-28
• Language: English
• Subtype: Poster