Notes

Lecturer: Matthew Cook

Feed-Forward Networks (FFNs) are not like the networks in the brain. In the feed-forward networks, the information moves in only one direction (forward) from the input nodes, through the hidden nodes (if they exist), and to the output nodes. There are no cycles in this network. Usually, people are referring to feed-forward networks when they talk about Artificial Neural Networks (ANNs). General structure:

• Multiple layers of neurons with a certain number of inputs and outputs.
• Every layer of nodes feeds the next layer with inputs.
• Each input in one layer is connected to all the inputs in the next layer.
• There is an input and an output layer with hidden layers in between.

A single unit, like a perceptron, can be seen as a feed-forward network. We can write down the connections of a FNN as a matrix of weights, so $w_{ij}$ is the weight from i to j. Why FNN are nice? Because we can think about functions that receive inputs and generate outputs. When we use FNN we know what we want to compute. We need to set the weights of the network in order to compute the function we want. The process of defining the weights is called learning or training. In feed-forward networks it is easy to evaluate each unit. The outputs are continuous functions of the input, which facilitates the optimization in case of wrong outputs. The training can be done using "training data": input/output pairs $\left( x_{i},d_{i} \right)$. Where $x_{i}$ is the input value and $d_{i}$ is the desired output. Then, we can define the error $E = \sum_{k}^{}\left( f\left( x_{k} \right) - d_{k} \right)^{2}$, where $f\left( x_{k} \right)$ is the output of the network. Differently of Hopfield Networks, we don't need continuous updates and we do not reevaluate units. FNN have the idea of a pipeline (unlike the brain). If a node on layer n in a FNN is connected to layer n + i with i > 1, this is still a FNN, however the most common structure is to connect nodes on layer n to nodes on layer n + 1. How can we change the weights to reduce the error? We can use gradient descent. We calculate all the $\frac{\partial E}{\partial w_{i}}$ in the network, easily, by starting at the end and then walking backwards. This is known as Backpropagation. Training Process

1. Give an input to the network.
2. Calculate all the values in the network to produce the output.
3. Compare output with the desired output. This gives us the "error" for this input.
4. Update the weights. For example, use gradient descent to optimize the parameters to minimize the error.

Backpropagation and Error Function Backpropagation is the process of calculating the derivatives, using the chain rule, from the last layer (directly connected with the output, thus, with the loss function) to the first layer (connected with the inputs). This process can be seen as walking through the network in a backward manner.

• The inputs and desired outputs are given as $S = (x,d)^{1},\ldots,(x,d)^{l}$.
• The error function is given as $E(S) = \sum_{i}^{}{\frac{1}{2}\left\| y\left( x^{i} \right) - x^{i} \right\|^{2}}$.
• The output is a non-linear transformation $y = f(a)$.
• $f(a)$ is the activation function, which is usually a sigmoid function.
• The error function for a single training sample is $E(S) = \frac{1}{2}\left( f\left( x_{1}w_{1} + x_{2}w_{2} + w_{0} \right) - d \right)^{2}$.
• The output of a simple network is for example $y\left( x_{1},x_{2},x_{3} \right) = f\left( x_{1}w_{21} + f\left( x_{2}w_{11} + x_{3}w_{12} + w_{10} \right)w_{22} + w_{20} \right)$.
• $\frac{\partial E\left( w_{1},w_{2},w_{0} \right)}{\partial w_{1}} = (f(x_{1}w_{1} + x_{2}w_{2} + w_{0}) - d) \bullet f^{'}(x_{1}w_{1} + x_{2}w_{2} + w_{0}) - d) \bullet x_{1}$.
• $\frac{\partial E\left( w_{1},w_{2},w_{0} \right)}{\partial w_{2}} = (f(x_{1}w_{1} + x_{2}w_{2} + w_{0}) - d) \bullet f^{'}(x_{1}w_{1} + x_{2}w_{2} + w_{0}) - d) \bullet x_{2}$.
• $\frac{\partial E\left( w_{1},w_{2},w_{0} \right)}{\partial w_{0}} = (f(x_{1}w_{1} + x_{2}w_{2} + w_{0}) - d) \bullet f^{'}(x_{1}w_{1} + x_{2}w_{2} + w_{0}) - d)$.
• The error terms travel backwards through the network and get multiplied with the derivative of the activation function of that input. Multiple error terms can just be added up.
• The partial derivative of the error E term in relation to the weight w to be adjusted can be added to the weight in order to learn. An additional weighting factor can be added.

Gradient Descent Consider $E = \sum_{}^{}\left( f_{w}(x) - y_{i} \right)^{2}$, we want to adjust $\overrightarrow{w}$ (weights of the network) to minimize E. Gradient descent is the process of descending through the gradients (using the derivatives calculated with backpropagation), in this algorithm we try to reach the minimum of the loss function. This is an iterative process.

Generally, the iterative process is given by $\theta_{i_{new}} = \theta_{i} + \mathrm{\Delta}\theta_{i}$, where $\mathrm{\Delta}\theta_{i} = - \alpha\frac{\partial J(\theta)}{\partial\theta_{i}}$.

In the above picture: the left-most figure: $f_{w_{0}}(x) = \theta(wf + w\ldots),\frac{df}{dw} = 0 \rightarrow \theta$ is not a good threshold function. To know in what direction we should move to find out minima, we need to use a threshold function that is continuous and differentiable, like the one in the center figure. Right figure: $\frac{df}{dw} = \theta'$.

We haven't yet found biological mechanisms that would be similar to gradient descent in the brain.

Boltzmann Machines Boltzmann Machines were invented in 1985 but not by Boltzmann. The name is given because these units use a Boltzmann distribution in their sampling function. These units are similar to Hopfield Networks, however, they have a probability of being active. When updating a unit, we set its value to zero or one probabilistically, following a sampling function (see figure below). Boltzmann machines do not converge, they do not reach a stable state. It can be seen as a system for sampling. It's a way to do a random walking in the state space.

Sampling There are many types of distributions. When we want to get examples of these distributions, we need to sample from it. And giving some samples we can recover the distribution of the data.

Figure: Representations of activation function for Hopfield Networks (HN) and Boltzmann Machines.

We want the units to forget previous states so the sampling is not biased, i.e., not similar to previous ones, thus really "random". While this sampling is nice, it is not useful as a memory, i.e., we don't want to sample things randomly from our memory.

We can define a network where the units have a real-valued activity level $\left( a_{i} \in \lbrack 0,1\rbrack \right)$, and also we can make time continuous, so $\frac{\partial a_{i}}{\partial t} = \theta\left( \sum_{}^{}\ \right) - a_{i}$. Using units like this, we can make a feed-forward network.