\chapter{Introduction}\label{chap:intro}

What is intelligence? From the early musings of Plato and Aristotle on minds and reasoning to current debates on what defines real artificial intelligence, humans have pondered this question for millennia. Arguably, the most compelling example—and the ultimate benchmark—of intelligence is our own brain: a biological network of remarkable complexity that seamlessly integrates perception, action, and learning. The past century has witnessed remarkable advancements in understanding the brain: from Pavlov's pioneering conditioning studies \citep{pavlov1906scientific} to the advent of functional magnetic resonance imaging (fMRI) \citep{ogawa1990brain}; from the concept of action potentials \citep{bernstein1902untersuchungen} to how they are transmitted between neurons \citep{hodgkin1952quantitative}; from the definition of synapses \citep{sherrington2023integrative} to the foundational principles of synaptic transmission and learning \citep{dale1935pharmacology,hebb1949organization}—all contributing to a more thorough and detailed understanding of the brain's components and their harmonious interplay.

Despite these extraordinary discoveries, a central question remains elusive: how do these components collectively give rise to the diverse functionalities and representations observed in the brain? Historically, computational neuroscience has sought to address this question by constructing bottom-up models that simulate brain processes. These models range from abstract, high-level frameworks—often employing one or a few equations to capture general principles, such as the brain's tendency to represent sensory information with minimal activity \citep{barlow1961coding}—to detailed, low-level systems that demonstrate how complex functions emerge from networks of simple units, such as spiking neural networks \citep{vogels2011inhibitory}. However, a persistent challenge is that many high-level models lack detailed neural implementations of their computational principles, while many low-level models are not grounded in overarching theoretical frameworks. Predictive coding bridges this gap by integrating both perspectives. Rooted in the Bayesian Brain hypothesis, which posits that the brain constructs and optimizes a generative model of sensory information \citep{doya2007bayesian}, predictive coding not only provides a high-level computational principle but also offers biologically plausible neural circuit implementations \citep{bogacz2017tutorial}. Building on this foundation, this thesis extends the predictive coding framework to address three fundamental challenges faced by the brain: space, time, and memory—all while preserving the integrity of both the high-level principles and the low-level neural plausibility.

\section{Predictive coding}
Before stepping into the specificity, two key ideas that lay the foundation of predictive coding need to be introduced. First, the Bayesian Brain hypothesis, and second, the idea of artificial neurons and networks. 

\subsection{Bayesian Brain}
\begin{figure}[t]
    \centering
    \includegraphics[width=1\linewidth]{figs//chap1/bayesianbrain.pdf}
    \caption{\textbf{The Bayesian Brain hypothesis.} A: Schematic of Bayesian Brain, where aroma (sensory input) is generated from a cup of coffee and the brain tries to reverse-engineer this process via Bayesian inference. B: Abstraction of Bayesian Brain using random variables $\bfz$ and $\bfx$.}
    \label{fig:bayesian_brain}
\end{figure}

Imagine waking up in the morning and making your way into the kitchen. Gladly someone is making coffee. Before you enter the kitchen you can already smell the aroma of coffee and you think, ``Ah, there's coffee in the kitchen!". How does this perception of coffee takes place, even though you did not witness the coffee itself with your own eyes? Could it be some other drinks emitting the aroma? All we know is that there exists some true, but hidden ``cause'' (the coffee) that generates the sensory observation (the aroma). One possibility is that the perception of the coffee happens as the brain tries to reverse-engineer this generative process, by building up its own generative model to approximate the true one and infer the cause from the model (Fig.~\ref{fig:bayesian_brain}A). This is the essence of the Bayesian Brain hypothesis.

The concept of Bayesian Brain was first articulated by \citet{helmholtz1866concerning} and then formalized in computational neuroscience in multiple studies in the late 90s and early 2000s \citep{dayan1995helmholtz,lee2003hierarchical,knill2004bayesian}. Multiple lines of experimental research have provided evidence for this Bayesian inference perspective of perception, such as those on sensory cue integration \citep{jacobs1999optimal,knill2003humans,hillis2004slant} and those on perceptual biases from inappropriate priors \citep{weiss2002motion}. This Bayesian approach to neural computation has also been applied to data fitting, from fMRI data \citep{behrens2014mr,friston2003dynamic} to electrophysiological recordings \citep{beck2007probabilistic}. 

Formally, Bayesian Brain can be described using a latent variable model depicted in Fig.~\ref{fig:bayesian_brain}B \citep{bishop2006pattern}, where $\mathbf{z}$ represents a hidden, or latent cause (coffee) and $\mathbf{x}$ represents the observed sensory input (aroma) of the brain. The generative model is parameterized by $\theta$, which provides an abstraction of the exact model. Due to uncertainty, the model cannot be deterministic and thus probability distributions are needed to describe the variables $\mathbf{z}$ and $\mathbf{x}$. Specifically, the probability distribution of interest is:
\begin{equation}
    p_{\theta}(\mathbf{z} | \mathbf{x})
\end{equation}
which defines the conditional distribution of the latent $\mathbf{z}$ given the observations $\mathbf{x}$. In essence, this distribution describes the underlying cause given the observations. However, in the setting of the Bayesian Brain hypothesis, the observable direction of dependence is the opposite i.e., the observation $\mathbf{x}$ is generated from the cause $\mathbf{z}$. We thus need to resort to Bayes theorem \citep{bayes1763essay} to find an alternative expression for the above distribution:
\begin{eqnarray}
    p_{\theta}(\mathbf{z} | \mathbf{x}) = \frac{p_{\theta}(\mathbf{x}|\mathbf{z})p_{\theta}(\mathbf{z})}{p_{\theta}(\mathbf{x})}
\end{eqnarray}
where $p_{\theta}(\mathbf{x}|\mathbf{z})$ represents the generative process, which is also referred to as the likelihood. $p_{\theta}(\mathbf{z})$ is a prior distribution on $\mathbf{z}$, and $p_{\theta}(\mathbf{x})$ is the marginal likelihood of the observed data, also known as ``evidence''. In this setting, the term $p_{\theta}(\mathbf{z} | \mathbf{x})$ is called the posterior distribution. Although the generative process and the prior are usually available under certain assumptions, the evidence is not, as it requires us to evaluate the intractable integral over all latent variables:
\begin{eqnarray}
    p_{\theta}(\mathbf{x}) = \int_{\mathbf{z}} p_{\theta}(\mathbf{x},\mathbf{z})d\mathbf{z} = \int_{\mathbf{z}} p_{\theta}(\mathbf{x} | \mathbf{z})p_{\theta}(\mathbf{z})d\mathbf{z}
\end{eqnarray}

This conundrum thus requires us to seek an alternative method to obtain the posterior. Although methods such as those based on sampling exist \citep{van2018simple}, we focus on an approximation approach called variation inference here.

\paragraph{Variational inference} One way of looking at variational inference is from an approximation perspective: if we cannot model $p_{\theta}(\mathbf{z} | \mathbf{x})$ exactly, can we find another distribution, say $q(\mathbf{z})$, to approximate it? To make the approximation as accurate as possible, we need to minimize a dissimilarity measure between $p_{\theta}(\mathbf{z} | \mathbf{x})$ and $q(\bfz)$. The standard measure is the Kullback-Leibler (KL) divergence, $\KL$ \citep{kullback1951information}, which is defined as follows:
\begin{eqnarray}\label{eq:kl}
    \KL(q(\mathbf{z}) || p_{\theta}(\mathbf{z} | \mathbf{x})) = \mathbb{E}_q[\log q(\mathbf{z}) - \log p_{\theta}(\mathbf{z}|\mathbf{x})]
\end{eqnarray}

It can be further written as (with Bayes Theorem):
\begin{eqnarray}
\begin{aligned}
    \KL(q(\mathbf{z}) || p_{\theta}(\mathbf{z} | \mathbf{x})) &= \mathbb{E}_q \left[\log q(\mathbf{z}) - \log \frac{p_{\theta}(\mathbf{z},\mathbf{x})}{p_{\theta}(\mathbf{x})}\right] \\&= \mathbb{E}_q[\log q(\mathbf{z}) - \log p_{\theta}(\mathbf{z},\mathbf{x})] + \log p_{\theta}(\mathbf{x}) \\ &= \mathcal{F} + \log p_{\theta}(\mathbf{x})
\end{aligned}
\end{eqnarray}
where $\mathcal{F} := \mathbb{E}_q[\log q(\mathbf{z}) - \log p_{\theta}(\mathbf{z},\mathbf{x})]$ is the variational free energy \citep{Feynman1972, friston2003learning,friston2005theory}. The term $-\mathcal{F}$ is also known as the evidence lower bound (ELBO) in machine learning, as the KL divergence is non-negative and thus $-\mathcal{F}$ serves as a lower bound for the ($\log$) evidence $\log p_{\theta}(\mathbf{x})$.

Note that to minimize the KL divergence as a function of $q(\mathbf{z})$, we only need to minimize the free energy $\mathcal{F}$ as $\log p_{\theta}(\mathbf{x})$ is not a function of $q(\mathbf{z})$. The minimization of $\mathcal{F}$, or the maximization of $-\mathcal{F}$, is often performed through the Expectation-maximization (EM) algorithm \citep{dempster1977maximum}, where $\mathcal{F}$ is first minimized with respect to $q(\mathbf{z})$ and then minimized with respect to the parameters $\theta$. As will be shown later, computations of predictive coding follows exactly this alternating process.

\subsection{Artificial neurons and networks}
\begin{figure}[t]
    \centering
    \includegraphics[width=1\linewidth]{figs//chap1/neurons.pdf}
    \caption{\textbf{Biological and artificial neurons.} A: Biological neurons and how they transmit signals. B: Artificial neurons and synapses as an abstraction of biological ones. C: A network of artificial neurons connected hierarchically.}
    \label{fig:neurons}
\end{figure}
The idea of artificial neurons and networks, as opposed to biological ones, is core to the low-level understanding of predictive coding. In particular, we focus on the neural network model where a neuron is assumed to receive a vector of inputs linearly weighted by synapses (weights) connecting each input neuron to the current neuron, perform a sum of the weighted inputs and then output the thresholded sum \citep{rosenblatt1958perceptron}. This process can be mapped to a biological neuron in Fig~\ref{fig:neurons}A where the dendrites of a neuron receive inputs from a pre-synaptic neurons, and the soma performs the summation function. Once the soma reaches a certain potential, the axon will transmit the signal to its terminal connecting another neuron via synapses. Artificial neurons provide an abstraction of this process, where specific architectural components of a neuron, including dendrites, soma and axon, are included in a computing unit in Fig~\ref{fig:neurons}B.

Formally, assuming $d_{in}$ pre-synaptic neurons providing an input vector $\mathbf{x}\in\mathbb{R}^{d_{in}}$, and $d_{out}$ post-synaptic neurons receiving the input and performing the thresholded linear summation and outputing $\mathbf{y}\in\mathbb{R}^{d_{out}}$. The synaptic weight is a $d_{out}\times d_{in}$ matrix $\mathbf{W}$. The output follows:
\begin{eqnarray}
    \mathbf{y} = \mathbf{W}f(\mathbf{x})
\end{eqnarray}
or if we look at individual neurons $i=1,...,d_{out}$:
\begin{eqnarray}\label{eq:ltu}
    y_i = \sum_{j=1}^{d_{in}}W_{ij}f(x_j)
\end{eqnarray}
where $f$ is the threshold function, which essentially provides a non-linear transformation of the inputs. Depending on the connectivity pattern, this network can either be fully feedforward (Fig~\ref{fig:neurons}C), which has been shown to have the capability of approximating any functions \citep{hornik1989multilayer}, or recurrent e.g., a Hopfield Network \citep{hopfield1982neural}, which can perform pattern completion/associative memory tasks. Eq.~\ref{eq:ltu} can also be described using a dynamical equation:
\begin{eqnarray}
    \dot{y}_i = -y_i + \sum_{j=1}^{d_{in}}W_{ij}f(x_j)
\end{eqnarray}
which has its equilibrium equating Eq.~\ref{eq:ltu} and captures the dynamical nature of neurons.

\subsection{Implementing Bayesian Brain in neural networks}
We are now ready to move on to predictive coding, which implements the Bayesian Brain hypothesis into a network of artificial neurons. On the basis of Bayesian Brain and the free energy principle, predictive coding makes the following two assumptions:
\begin{itemize}
    \item[1.] The variational approximation $q(\mathbf{z})$ follows a delta distribution with density concentrated on $\boldsymbol{\phi}$ i.e., $q(\mathbf{z}) = \delta(\mathbf{z} - \boldsymbol{\phi})$. Furthermore, the center $\boldsymbol{\phi}$ has a Gaussian prior $\boldsymbol{\phi} \sim \mathcal{N}(\boldsymbol{\mu},\bm{\Sigma_\phi})$;
    \item[2.] The generative process $p_{\theta}(\mathbf{x}|\mathbf{z})$ follows a Gaussian distribution $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\Theta}f(\mathbf{z}),\boldsymbol{\Sigma}_\bfx)$,
\end{itemize}
which enable us to re-write the free energy $\mathcal{F}$ as:
\begin{eqnarray}
\begin{aligned}
    \mathcal{F} &= \int_{\mathbf{z}} q(\bfz)\log q(\bfz)d\bfz - \int_{\mathbf{z}} q(\bfz) \log p_{\theta}(\bfx,\bfz)d\bfz \\&= -\log p_{\theta}(\bfx, \boldsymbol{\phi}) + C \\&= -\log p_{\theta}(\bfx | \boldsymbol{\phi}) - \log p_{\theta} (\boldsymbol{\phi}) + C \\&= \frac{1}{2}(\bfx - \bm{\Theta}f(\bm{\phi}))^{\top}\bm{\Sigma}_\bfx^{-1} (\bfx - \bm{\Theta}f(\bm{\phi})) + \frac{1}{2}(\bm{\phi} - \bm{\mu})^{\top}\bm{\Sigma}^{-1}_{\bm{\phi}}(\bm{\phi} - \bm{\mu}) \\&+ \frac{1}{2}\log |\bm{\Sigma_x}| + \frac{1}{2}\log |\bm{\Sigma_\phi}| + C
\end{aligned}
\end{eqnarray}
where from the first to the second line we use the fact that for a delta function $\delta(\mathbf{z} - \boldsymbol{\phi})$, the integral $\int_{\mathbf{z}} q(\bfz) \log p_{\theta}(\bfx,\bfz)d\bfz$ is equal to $\log p_{\theta}(\bfx, \boldsymbol{\phi})$, and that $\int_{\mathbf{z}} q(\bfz)\log q(\bfz)d\bfz$ is independent of $\bm{\phi}$ and can thus be absorbed in a constant $C$. This expression of the free energy can be regarded as the sum of the (weighted and squared) prediction errors $\bfx - \bm{\Theta}f(\bm{\phi})$ and $\bm{\phi} - \bm{\mu}$, thus the name ``predictive coding''. Intuitively, in the predictive coding framework, neurons in the brain generate predictions of the sensory inputs, resulting in prediction errors. The objective of the brain, which leads to its synaptic and neuronal dynamics, is to minimize the predictions errors. When the prediction errors are close to $0$, $\bm{\phi}$ is predictive of the sensory observation $\bfx$, thus forming a ``good'' representation of the sensory input.

Using the EM algorithm to minimize the free energy (and thus prediction errors) consists of first minimizing $\mathcal{F}$ with respect to $\bm{\phi}$, which defines $q(\bfz)$, via gradient descent:
\begin{eqnarray}
    \dot{\bm{\phi}} \propto -\frac{\partial \mathcal{F}}{\partial {\bm{\phi}}} = -\bm{\Sigma}^{-1}_{\bm{\phi}}(\bm{\phi} - \bm{\mu}) + f'(\bm{\phi})\odot \bm{\Theta}^{\top} \bm{\Sigma}_\bfx^{-1} (\bfx - \bm{\Theta}f(\bm{\phi}))
\end{eqnarray}
where $\odot$ is the element-wise product between vectors. If we define $\bm{\epsilon}_\bfx := \bm{\Sigma}_\bfx^{-1} (\bfx - \bm{\Theta}f(\bm{\phi}))$ and $\bm{\epsilon_\phi} := \bm{\Sigma}^{-1}_{\bm{\phi}}(\bm{\phi} - \bm{\mu})$, this becomes:
\begin{eqnarray}\label{eq:hpcn-inference}
    \dot{\bm{\phi}} \propto - \bm{\epsilon_\phi} + f'(\bm{\phi})\odot \bm{\Theta}^{\top}\bm{\epsilon}_\bfx
\end{eqnarray}

The second step is to minimize $\mathcal{F}$ with respect to the parameters $\theta = \{\bm{\Theta}, \bm{\mu}, \bm{\Sigma}_\bfx, \bm{\Sigma_{\bm{\phi}}}\}$, also via gradient descent:
\begin{eqnarray}\label{eq:1layer-learning}
    \Delta\bm{\Theta}\propto -\frac{\partial \mathcal{F}}{\partial \bm{\Theta}} = \bm{\epsilon}_\bfx f(\bm{\phi})^{\top}; \quad \Delta\bm{\mu} \propto -\frac{\partial \mathcal{F}}{\partial \bm{\mu}} = \bm{\epsilon_\phi}
\end{eqnarray}
\begin{eqnarray}\label{eq:cov-learning}
    \Delta\bm{\Sigma}_\bfx \propto -\frac{\partial \mathcal{F}}{\partial \bm{\Sigma}_\bfx} = \frac{1}{2}(-\bm{\Sigma}_\bfx^{-1} + \bm{\epsilon}_\bfx \bm{\epsilon}_\bfx^{\top});\quad \Delta\bm{\Sigma}_{\bm{\phi}} \propto -\frac{\partial \mathcal{F}}{\partial \bm{\Sigma}_{\bm{\phi}}} = \frac{1}{2}(-\bm{\Sigma}_{\bm{\phi}}^{-1} + \bm{\epsilon}_{\bm{\phi}} \bm{\epsilon}_{\bm{\phi}}^{\top})
\end{eqnarray}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.9\linewidth]{figs//chap1/hierarchical.pdf}
    \caption{\textbf{Network implementation of predictive coding.} A: A simple, two-layer case. B: Extending predictive coding to a multi-layer architecture.}
    \label{fig:hpcn-implementation}
\end{figure}

\paragraph{Neural implementation} At this stage, the objective of free energy minimization has been translated into the dynamic equations above describing neural and synaptic activities. How may a neural network implement such computations? If we assume $\bm{\Sigma}_{\bm{\phi}}$ and $\bm{\Sigma}_\bfx$ are both identity matrices for simplicity (this will be discussed in more detail at later parts of this thesis), the computations above can be performed by the network shown in Fig.~\ref{fig:hpcn-implementation}A, where we implement $\bfx$, $\bm{\phi}$ and $\bm{\epsilon}$'s as neural activities, and $\bm{\Theta}$ and $\bm{\mu}$ as synaptic weights. Error neurons $\bm{\epsilon}$ (circles) will receive excitatory inputs (red arrows) from their corresponding value neurons (triangles)\footnote{Note that throughout this thesis, ``value'' neuron is a concept completely unrelated to ``value'' functions in reinforcement learning. Rather, it represents the value of neural activities, to compare to the error of neural activities.}, and inhibitory inputs from top-down paths (left-to-right in the schematic) to compute the prediction mismatch. The neural dynamics $\dot{\bm{\phi}}$ is driven by excitatory inputs from $\bm{\epsilon}_\bfx$ and inhibitory inputs from $\bm{\epsilon}_{\bm{\phi}}$. Importantly, all neurons and synapses in this network use only locally available information, and in particular, the learning rules (Eq.~\ref{eq:1layer-learning}) are Hebbian i.e., they are products of pre- and post-synaptic activities \citep{hebb1949organization}. The nonlinear transformation $f$ can also be naturally implemented in this network (see \citet{whittington2017approximation} for details).

A characteristic feature of the network implementation above is the separation of value and error neurons, which enables the local computations. Value neurons $\bm{\phi}$ sends top-down prediction to the sensory level, while error neurons $\bm{\epsilon}_\bfx$ sends bottom-up error signals that corrects the value neurons to reduce predictions errors over time. There is abundant evidence for the existence of error signals in neocortex \citep{keller2018predictive} and other regions such as the hippocampus \citep{ku2021contributions}. Although the exact neuron-level encoding of prediction errors is less generic and restricted to areas such as dopaminergic reward prediction errors \citep{schultz1998predictive}, it is also possible to implement the error signals into dendrites of pyramidal neurons to avoid the error-encoding neurons in predictive coding \citep{mikulasch2023error}. This dendritic approach has been shown to match backpropagation in training feedforward neural networks \citep{sacramento2018dendritic,greedy2022single,whittington2019theories} will be adopted throughout this thesis. In addition, the error neurons in this network implementation can carry both positive and negative values, which can be considered as separate neuron populations signaling ``more'' and ``less'', as discussed in \citep{keller2018predictive}.

\paragraph{Learning covariance matrices} The above network implementation did not account for the learning of the covariance matrices $\bm{\Sigma_\phi}$ and $\bm{\Sigma}_\bfx$ (Eqs.~\ref{eq:cov-learning}).  The matrix inverse poses challenges for implementing the plasticity computation within a neural network: Computationally, inverting a (large) matrix is inefficient and unstable; Biologically, computing one element of an inverted matrix requires accessing all the elements of the original matrix, making the computation non-local and non-Hebbian. Therefore, an external storage mechanism is needed to store all the synaptic weights to update one single synapse. Although \citet{bogacz2017tutorial} has proposed a network that can implement the learning of covariance matrices locally, the implementation requires additional interneurons complicating the circuit. In Chapter \ref{chap:am} of this thesis, I will propose an alternative way of implementation based on a reparameterization of the covariance matrices. For introductory purposes, the remainder of this chapter will focus on the case where both $\bm{\Sigma_\phi}$ and $\bm{\Sigma}_\bfx$ are identity matrices.

\paragraph{Multi-layer predictive coding} In the description of predictive coding above, $\bfx$ denotes the lowest-level sensory input. However, the theory can also be extended to multiple levels $\bfx^{(l)},l=1,...,L$, to account for the multi-layer structure of the neocortex. If we denote the lowest-level sensory input as $\bfx^{(1)}$, and the topmost latent cause as $\bfx^{(L)}$, the free energy of a multi-layer predictive coding model can be derived as:
\begin{eqnarray}
\begin{aligned}
    \mathcal{F} &= -\log p_{\theta}(\bfx^{(1)}, \bfx^{(2)},...,\bfx^{(L)})
    \\&= -\log \left[\prod_{l=1}^{L-1} p_{\theta}(\bfx^{(l)} | \bfx^{(l+1)})\right]p_{\theta}(\bfx^{(L)})
    \\&= -\sum_{l=1}^{L-1}\log p_{\theta}(\bfx^{(l)} | \bfx^{(l+1)}) - \log p_{\theta}(\bfx^{(L)})
\end{aligned}
\end{eqnarray}

Assuming $\bfx^{(l)} | \bfx^{(l+1)}\sim \mathcal{N}(\bm{\Theta}^{(l+1)} f(\bfx^{(l+1)}),\mathbf{I})$ and $\bfx^{(L)}\sim \mathcal{N}(\bm{\mu},\mathbf{I})$, where $\mathbf{I}$ denotes the identity covariance matrix, and $\bm{\Theta}^{(l+1)}$ denotes the weights connecting the $l+1$th layer and the $l$th layer, the free energy can be further expressed as:
\begin{eqnarray}
\begin{aligned}
    \mathcal{F} &= \sum_{l=1}^{L-1} \frac{1}{2}\Vert \bfx^{(l)} - \bm{\Theta}^{(l+1)} f(\bfx^{(l+1)})\Vert_2^2 + \frac{1}{2}\Vert \bfx^{(L)} - \bm{\mu}\Vert_2^2 + C
    \\&= \frac{1}{2}\sum_{l=1}^L \Vert \bm{\epsilon}^{(l)}\Vert_2^2 + C
\end{aligned}
\end{eqnarray}
where we define $\bm{\epsilon}^{(l)}:= \bfx^{(l)} - \bm{\Theta}^{(l+1)} f(\bfx^{(l+1)}) , \forall l\neq L$ and $\bm{\epsilon}^{(L)}:=\bfx^{(L)} - \bm{\mu}$. In essence, the free energy can be viewed as the sum of squared prediction errors at each layer, where the predictions are parameterized by first applying a nonlinear transformation on the higher layer's activity, and then applying a linear transformation by $\bm{\Theta}^{(l+1)}$. The objective of a predictive coding model is to minimize the prediction errors at all layers. This objective is achieved by minimizing $\mathcal{F}$ with respect to both $\bfx^{(l)}$ (neural activities) and $\theta = \{\bm{\Theta}^{(l)}, \bm{\mu}\}$, following gradient descent:
\begin{eqnarray}
    \dot{\bfx}^{(l)}\propto -\frac{\partial \mathcal{F}}{\partial \bfx^{(l)}} = -\bm{\epsilon}^{(l)} + f'(\bm{\bfx^{(l)}})\odot (\bm{\Theta}^{(l)})^{\top}\bm{\epsilon}^{(l-1)}
\end{eqnarray}
\begin{eqnarray}
    \Delta\bm{\Theta}^{(l)} \propto -\frac{\partial \mathcal{F}}{\partial \bm{\Theta}^{(l)}} = \bm{\epsilon}^{(l-1)}f(\bfx^{(l)})^{\top};\quad \Delta\bm{\mu}\propto -\frac{\partial\mathcal{F}}{\partial \bm{\mu}} = \bm{\epsilon}^{(L)}
\end{eqnarray}
Similar to the two-layer case, these computations can also be implemented in a network with local dynamics and Hebbian plasticity, shown in Fig.~\ref{fig:hpcn-implementation}B. 

\paragraph{Modeling the brain with predictive coding}
Predictive coding has been primarily used to model the visual system. For example, \citet{srinivasan1982predictive} argued that predictive coding provides a natural explanation for inhibition in the retina. In their seminal work, \citet{rao1999predictive} modeled the extra-classical receptive-field effects in the visual cortex using predictive coding, a phenomenon postulated in the theoretical work by \citet{mumford1992computational}. They also found that the model developed Gabor-like visual filters observed in the brain \citep{hubel1962receptive} following a sparse constraint on the latent activities, similar to what was found with the sparse coding theory \citep{olshausen1996emergence}. Using fMRI, \citet{murray2002shape} found that grouping individual elements into shapes increased activities in higher visual areas while decreased activities in early areas, providing experimental evidence for the subtraction of top-down signals in lower areas in predictive coding. Predictive coding has also been successfully used to explain V4-prefontal cortex interaction in occluded visual scenes \citep{choi2018predictive}, repetition suppression \citep{auksztulewicz2016repetition}, bistable perception \citep{hohwy2008predictive,weilnhammer2017predictive}, illusory motions \citep{watanabe2018illusory} and retinal stabilization \citep{millidge2019fixational}.

Features from other brain regions may also be explained using the predictive coding theory. For example, in the auditory domain, predictive coding provides a mechanistic explanation for the mismatch negativity phenomenon, where the brain responds to a abnormal auditory event \citep{garrido2009mismatch}. In the domain of motor control, predictive coding has been used to explain the mirror neurons i.e., neurons that fire not only when a motion is executed but also when it is observed \citep{kilner2007predictive}. The modeling of motor control with predictive coding can also be extended to the modeling of social interactions \citep{wolpert2003unifying}. A recent theory has suggested that the hippocampo-neocortical interactions can be explained under a predictive coding account, and in particular, predictive coding underlies both prediction and memory in the hippocampus \citep{barron2020prediction}. This theory was later verified in simulations, where a predictive coding model successfully performed an associative memory task \citep{salvatori2021associative}. Many studies have also focused on matching computations of predictive to cortical microcircuits \citep{bastos2012canonical,keller2018predictive}. Although these studies have established behavioral and functional modeling using predictive coding in these other regions, the representational modeling capability of predictive coding has been limited to the visual cortex. This gap will be addressed in this thesis by extending the representational modeling of predictive coding to the medial entorhinal cortex in Chapter \ref{chap:grid}. 

\paragraph{Relationship to machine learning}
In machine learning and statistics, minimizing the free energy $\mathcal{F} = \mathbb{E}_q[\log q(\mathbf{z}) - \log p_{\theta}(\mathbf{z},\mathbf{x})]$ is an important problem as its negative value $-\mathcal{F}$ serves as a lower bound for the evidence $\log p_{\theta}(\bfx)$, which is ultimately the value that any model $\theta$ of the data $\bfx$ wants to maximize. Historically, many algorithms have been proposed to efficiently perform free energy minimization, such as the incremental EM algorithm \citep{neal1998view} and the Helmholtz Machine \citep{dayan1995helmholtz}. The former algorithm has recently been incorporated into predictive coding for improved efficiency \citep{salvatori2022incremental}, although the optimization process still involves solving a dynamical equation iteratively (Eq.~\ref{eq:hpcn-inference}). The latter model uses a wake-sleep algorithm to perform optimization, which, however, does not guarantee minimization of the exact free energy. Modern machine learning has focused on circumventing the computationally expensive inference of $q(\bfz)$ i.e., the E step in the EM algorithm. For example, variational autoencoder \citep{kingma2013auto} amortized the inference with a parameterized model (e.g., a neural network), essentially asking the model to ``learn to infer''. Such models have achieved great success in generating realistic but unseen images given a large set of training data. Although classical predictive coding lacks such capability due to the delta distribution assumption on $q(\bfz)$, a recent study has successfully extended predictive coding for this task with Monte Carlo sampling \citep{oliviers2024learning}.

As described above, predictive coding is a training algorithm for multi-layer neural networks. This connects it to backpropagation \citep{rumelhart1986learning}, the training algorithm underlying almost all mainstream deep learning architectures \citep{lecun2015deep}. The extraordinary ability of backpropagation to train neural networks once raised great interests in adopting it to model learning in the brain. However, many aspects of the algorithm are unrealistic from a biological perspective \citep{crick1989recent}. Most notably, the weight updates in backpropagation are non-local i.e., to update a single synapse, backpropagation requires access to other weights in the network. On the other hand, predictive coding uses Hebbian plasticity for weight updates. Strikingly, \citet{whittington2017approximation} showed that, under certain conditions, predictive coding can approximate backpropagation when training a multi-layer neural network. It is then shown that this approximation still holds along arbitrary computational graphs beyond a multi-layer structure \citep{millidge2020predictive}, and a version of predictive coding that implements backpropagation exactly was also proposed \citep{song2020can}. These results have provided evidence for a possible implementation of backpropagation in the brain using dynamics of predictive coding. Interestingly, recent studies have also revealed certain machine learning tasks, which are also the ones usually faced by biological organisms, where predictive coding outperforms backpropagation e.g., learning with small training sets \citep{song2024inferring}.

\section{Thesis research and overview}

Building on the foundation of predictive coding, this thesis explores three relatively unexamined or underdeveloped aspects of the theory, thereby extending its application to broader functions and representations of the brain. These three aspects are \textit{space, time, and memory}. Note that the thesis will not follow this specific order of the three aspects, as certain aspects may be revisited in the discussion of others. 

\subsection{Memory}
Theoretical \citep{barron2020prediction} and modeling \citep{salvatori2021associative} studies applying predictive coding to the hippocampus and its role in memory have provided initial evidence that predictive coding can effectively model this crucial brain function. However, a key anatomical feature of the hippocampus—its recurrent connectivity—is absent from the current model \citep{salvatori2021associative}. Many computational models, such as the Hopfield Network \citep{hopfield1982neural}, capture the recurrent nature of the hippocampal network and rely on these recurrent connections to perform memory tasks. The next part of this thesis (Chapter \ref{chap:am}) will explore a recurrent formulation of predictive coding, in which a single layer of neurons performs associative memory by predicting one another.

A natural candidate for recurrent predictive coding is a model incorporating covariance matrices ($\bm{\Sigma}$), as proposed in the recurrent network implementations of predictive coding by \citet{friston2003learning} and \citet{bogacz2017tutorial}, where recurrent connections encode $\bm{\Sigma}$. Furthermore, covariance and correlation naturally relate to associative memory \citep{kohonen1972correlation}. We will demonstrate that this formulation of recurrent predictive coding can indeed perform associative memory but suffers from biological implausibility and numerical instability due to the matrix inversion required in the covariance learning rule (as discussed earlier). To address these limitations, we will propose an alternative formulation of recurrent predictive coding that is both more biologically plausible and numerically stable. Notably, this revised model is mathematically equivalent to the classical recurrent predictive coding model in associative memory, effectively serving as a reparameterization. Finally, we will establish a connection between recurrent predictive coding and Hopfield Networks, bridging two of the most influential theories of neural computation.

\subsection{Time}
The word ``recurrent'' is naturally reminiscent of temporal processing. Although the aforementioned ``recurrent'' predictive coding concerns only a static layer of neurons recurrently connected with each other, Chapter \ref{chap:time} of the thesis will discuss another form of recurrence involving time. To avoid confusion, we will term it as ``temporal'' predictive coding. Although earlier studies have attempted to extend predictive coding to the temporal and dynamical domain \citep{rao1997dynamic,friston2003dynamic,friston2009predictive}, none have inherited the simple and plausible implementation of hierarchical predictive coding (Fig.~\ref{fig:hpcn-implementation}) while incorporating the neural components that enable temporal processing. We will present a novel formulation of temporal predictive coding that retains the plausible network implementation. We will demonstrate that this formulation is rooted in the first principle of Bayesian filtering \citep{stengel1986stochastic} and can thus be regarded as an approximation of Kalman filters \citep{kalman1960new}. Mirroring the way that \citet{rao1999predictive} followed to apply predictive coding to natural images, we will also apply temporal predictive coding to natural \emph{movies}. Intriguingly, we found not only Gabor-like edge detectors resembling V1 simple cells \citep{hubel1962receptive}, but also motion-detecting neurons \citep{ohzawa1996encoding}.

It is in Chapter \ref{chap:seqmem} that we will return to memory, although this time we will explore \emph{sequential} memory. In contrast to memories of static patterns, such as images, sequential memories memorize dynamic patterns, such as movies. We will show that temporal predictive coding is perfectly suitable for the sequential memory task. We will provide a mathematical analysis of how temporal predictive coding performs sequential memory, deriving an analytical expression for the retrieved sequential patterns. Importantly, this analytical expression will relate temporal predictive coding to a dynamical extension of Hopfield Networks: Asymmetric Hopfield Network \citep{sompolinsky1986temporal}, thus further bridging these two influential theories. The expression will also reveal an implicit whitening operation happening in temporal predictive coding, which has been proposed to happen in the brain \citep{carandini2012normalization} and gives it advantages over Asymmetric Hopfield Networks in sequential memory. Furthermore, we will go beyond simple memory retrieval and demonstrate the context-learning and generalization capabilities of temporal predictive coding. We will show that temporal predictive coding will develop representations of time steps in its latent activities, essentially signaling ``where I am'' within a sequential input.

\subsection{Space}
This final part, Chapter \ref{chap:grid}, of the thesis will focus on an aspect often intertwined with time: space. In particular, it will focus on an intriguing representation of space in the medial entorhinal cortex of the brain: grid cells \citep{hafting2005microstructure}, and try to explain the emergence of grid cells from the lens of predictive coding. With their unique periodic firing pattern over space that forms a hexagonal grid, grid cells serve as a metric system that provides contextual spatial information to the brain. This contrasts them to cells signaling individual locations i.e., place cells \citep{o1976place}. Following the earlier discussion of temporal predictive coding's context-learning capability, the motivation for this line of research is straightforward: If temporal predictive coding can represent temporal context, can it also develop spatial contextual representations like grid cells?

We will show that if temporal predictive coding is trained to perform path integration i.e., estimating one's current location based on velocity inputs, its latent neurons will form hexagonal firing centers over space, resembling grid cells. We will also compare temporal predictive coding with another model of grid cells based on recurrent neural networks \citep{sorscher2023unified}. Recurrent neural networks are trained with backpropagation through time \citep{werbos1990backpropagation}, which is implausible due to similar reasons for the implausibility of backpropagation discussed earlier. We will show that temporal predictive coding, which is biologically more plausible, is equivalent to a special case of backpropagation through time. This theory thus extends the relationship between predictive coding and backpropagation \citep{whittington2017approximation} to the temporal learning domain.


