Binarized neural networks converge toward algorithmic simplicity: empirical support for the learning-as-compression hypothesis

Journal article

Understanding and controlling the complexity of neural networks is a central challenge in machine learning, with implications for generalization, optimization, and model capacity. While most approaches rely on entropy-based loss functions and statistical metrics, these measures often fail to capture deeper, causally relevant algorithmic regularities embedded in network structure. We propose a shift toward algorithmic information theory, using binarized neural networks (BNNs) as a first proxy. Grounded in algorithmic probability (AP) and the universal distribution it defines, our approach characterizes learning dynamics through a formal, causally grounded lens. We apply the Block Decomposition Method (BDM), a scalable approximation of algorithmic complexity based on AP, and demonstrate that it more closely tracks structural changes during training than entropy, generally exhibiting stronger correlations with training loss across a wide range of architectures, datasets, and randomized training runs. These results support the view of training in BNNs as a process of algorithmic compression, where learning corresponds to the progressive internalization of structured regularities. In doing so, our work offers a principled estimate of learning progression and suggests a framework for complexity-aware learning and regularization, grounded in first principles from information theory, complexity, and computability.

Authors: Sakabe, EY; Abrahão, FS; Simões, A; Colombini, E; Costa, P

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Frontiers Media
• Journal: Frontiers in Computational Neuroscience
• Volume: 20
• Article number: 1791546
• Publication date: 2026-05-29
• Acceptance date: 2026-05-05
• DOI: 10.3389/fncom.2026.1791546
• EISSN: 1662-5188
• ISSN: 1662-5188
• Language: English
• Keywords: learning as compression; Block Decomposition Method; algorithmic information theory; binarized neural networks; Shannon entropy