Notes

Content of the Lecture

• Unstructured Recurrence
  • Circuit-level Recurrence
    • Anatomical Evidence
    • Functional Evidence
  • Training Recurrent Nets
    • Backpropagation Through Time
    • Reservoir Computing (Computation and Dynamics)
    • Biological Learning Rules (Hopfield Nets)
    • SORN
• Hierarchical Recurrence
  • Evidence in Primates
    • Processing Time as Evidence
  • LSTMs
    • Implementation
    • Functions
  • Potential Uses
    • Predictive Coding
    • Training Feedforwardness
  • Challenges

RNNs in the Brain - Circuit-Level Recurrence - Anatomical & Functional Evidence

Motivation: Why Is It So Important to Understand RNN Learning?

• The mammalian cortex is highly recurrent - it will help us to better understand the brain.
• The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: sequences in the input, the output, or in the most general case both. Thus, better understanding RNNs might help us to develop new, powerful RNN algorithms that are able to learn long and complex sequences.
• In programming terms be interpreted as running a fixed program with certain inputs and some internal variables. Viewed this way, RNNs essentially describe programs. In fact, it is known that RNNs are Turing-Complete in the sense that they can be used to simulate arbitrary programs (with proper weights).
• RNNs perform exceptionally well in language-modelling, the task of predicting the probability of the next word in a sequence.

RNNs in the Brain There are claims that networks of the AlexNet type successfully predict properties of neurons in visual cortex. Thus, one natural question arises: how similar is an ultra-deep residual network to the primate cortex? A notable difference is the depth. While a residual network has many as 1202 layers, biological systems seem to have two orders of magnitude less, if we make the customary assumption that a layer in the NN architecture corresponds to a cortical area. In fact, there are about half a dozen areas in the ventral stream of visual cortex from the retina to the Inferior Temporal Cortex. Notice that it takes in the order of 10ms for neural activity to propagate from one area to another one (remember that spiking activity of cortical neurons is usually well below 100Hz). The evolutionary advantage of having fewer layers is apparent: it supports rapid (100ms from image onset to meaningful information in IT neural population) visual recognition, which is a key ability of human and non-human primates. It is intriguingly possible to account for this discrepancy by taking into account recurrent connections within each visual area. Areas in visual cortex comprise six different layers with lateral and feedback connections, which are believed to mediate some attentional effects and even learning (such as backpropagation). "Unrolling" in time the recurrent computations carried out by the visual cortex provides an equivalent "ultra-deep" feedforward network, which might represent a more appropriate comparison with the state-of-the-art computer vision models.

Recurrent Projections in the Cat Brain Paper: "A Quantitative Map of the Circuit of Cat Primary Visual Cortex".

By mapping the circuit of cat primary visual cortex (V1), it is evident, that there are recurrent projections involved.

Anatomical Evidence for RNNs in the Rodent Brain Paper: "Distinct Timescales of Population Coding Across Cortex".

(Train a mice to turn left or right depending on sound location and Record neural activity in Auditory Cortex and Posterior Parietal Cortex). Similarly, it has been shown in rodents, that the communication between columns is organized by multiple highly specific horizontal projection patterns. Population coding is a method to represent stimuli by using the joint activities of a number of neurons. In population coding, each neuron has a distribution of responses over some set of inputs, and the responses of many neurons may be combined to determine some value about the inputs.

Anatomical Evidence for RNNs in the Primate Brain The brain has both a feed-forward structure and recurrent pathways. Information can get sent back from one area to a previous one or echo around the same area multiple times. Studies suggest this extra processing helps the brain interpret challenging visual information, such as objects that are occluded or viewed from unusual angles. A recent study found images that are difficult for a feed-forward model to classify but easy for humans and monkeys to interpret, although they take slightly longer to classify these challenging images than normal ones. This delay suggests that some recurrent processing is involved. The researchers then looked at how neural activity in the monkey's brain evolves as these images are processed. A benefit of convolutional neural networks is that the response of different layers in the model can be used to predict the response of neurons in different brain areas. The researchers found that the feed-forward model predicts the activity of neurons fairly well at early stages (up to 0.1s into the response) but struggles at later time points. When a convolutional neural network is not performing well, researchers in computer vision tend to add more layers to it, making it "deeper". The authors tested whether such deeper networks could better predict neural responses to their challenging images, under the assumption that a network with more layers, which computes over space, resemble recurrent pathways, which compute over time. These deeper networks were indeed better than the shallower model at predicting neural activity at later time points. Finally, the authors added recurrent connections to the structure of their original model and found that responses at later time points in the model better matched later time points in the data. Specifically, when recurrent connections were added to this "shallower" network, it predicted neural activity as well as the deeper model did. Overall, this work strongly suggests that recurrent processing is an important contributor to computation in the visual system.

In the image below: Both primates and feedforward DCNNs were tasked to identify which object is present in each test image (1320 images). Top: the stages in the primate ventral visual pathway (retina, LGN, V1, V2, V4, and the IT cortex), which is implicated in core object recognition. We can conceptualize each stage as rapidly transforming the representation of the image and ultimately yielding the primates' behavior (i.e., producing a behavioral report of which object was present). The blue arrows indicate the known anatomical feedforward projections from one area to the other. The red arrows indicate the known lateral and top-down recurrent connections. Bottom: a schematic of a similar pathway commonly present in DCNNs. These networks contain a series of convolutional and pooling layers with nonlinear transforms at each stage, followed by fully connected layers (which approximate macaque IT neural responses) that ultimately gives rise to the models' "behavior". Note that the DCNNs only have feedforward (blue) connections.

Functional Evidence Generate two models of neural activity incorporating any variable we can think of with a Generalized Linear Model (GLM). The predictors can be trained in isolation (uncoupled) or dependent on previous neuron activity (coupled). The Coupled model performs much better for PPC, ergo we assume the recurrence is important. For AC both perform similarly, but AC is less recurrent than PPC.

RNNs in Machine Learning & Back-Propagation Through Time

RNNs in Machine Learning

In the left figure: each rectangle is a vector and arrows represent functions (e.g., matrix multiply). Input vectors are in red, output vectors are in blue and green vectors hold the RNN's state. From left to right: (1) Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g., image classification). (2) Sequence output (e.g., image captioning takes an image and outputs a sentence of words). (3) Sequence input (e.g., sentiment analysis where a given sentence is classified as expressing positive or negative sentiment). (4) Sequence input and sequence output (e.g., Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French). (5) Synced sequence input and output (e.g., video classification where we wish to label each frame of the video). Notice that in every case are no pre specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like.

Recurrent Neural Networks (RNNs) add an interesting twist to basic neural networks. A vanilla neural network takes in a fixed size vector as input which limits its usage in situations that involve a "series" type input with no predetermined size. Recurrent nets allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both (A sequence means, that the elements can have dependency on each other and that the order matters!). A few examples that may make this more concrete, are shown in the previous figure. The size of the input or output sequence is flexible, i.e., does not change the architecture of the model. Each network state gets an indices for the sequence. Since the sequence is often related with time progression, the index is chosen to be t. The main difference in architecture compared to conventional ANNs is, that recurrent loops are allowed, i.e., inputs from previous layer states of the network. Looking at a one-to-one neural network with one hidden layer, we can write the output state y(t) and the hidden layer state h(t) as follows:

where we include the bias in the W matrix. If we want to display the network over all sequences graphically, i.e., the computational graph, we can unroll it as displayed below.

This gives us another perspective: for any fixed sequence length s, the unrolled recurrent network corresponds to a feedforward network with s hidden layers. The two main differences to a feedforward network is, that the inputs are processed and outputs produced in sequence, and that the same parameters are used for all layers/all time steps, i.e., the same functions U, V, W applied over all times steps (Not to be confused with all epochs).

Back-Propagation Through Time (BPTT) The unfolding shown in the figure above is the first step of a particular network training algorithm, which is called Back-Propagation Through Time (BPTT). The second step is applying our known backpropagation algorithm to the unrolled network to calculate all weight updates.

There are several drawbacks to BPTT:

• Costly Parameter Update: Especially for long sequences, the parameter update for a shallow layer is the same as updating a parameter in an extremely deep feedforward network. One way to fix this is using a truncated BPTT algorithm. It processes the sequence one timestep at a time, and every k1 timesteps, it runs BPTT for k2 timesteps, so a parameter update can be cheap if k2 is small. Consequently, its hidden states have been exposed to many timesteps and so may contain useful information about the far past, which would be opportunistically exploited.
• Exploding Gradients: The gradients coming from the deeper layers have to go through continuous matrix multiplications because of the chain rule, and as they approach the earlier layers. If they have large value (>1) they get larger and eventually blow up and crash the model (NaN-values!). This can be solved by gradient clipping: which places a predefined threshold on the gradients to prevent it from getting too large. Note that this only changes the length, and not the direction of the gradients.
• Vanishing Gradients: A similar problem arises if the gradients have small values (<1). They will shrink exponentially until they vanish and make it impossible for the model to learn. This issue cannot be solved as simple; hence it requires to use shorter sequences or to make fundamental change in the RNN architecture.

RNNs in Theoretical Neuroscience - Hopfield Networks, Reservoir Computing & Self-Organizing Recurrent Networks (SORN)

RNNs in Theoretical Neuroscience Hopfield Network A Hopfield network is a form of recurrent artificial neural network popularized by John Hopfield in 1982, but described earlier by Little in 1974. Hopfield nets serve as content-addressable ("associative") memory systems with binary threshold nodes. They are guaranteed to converge to a local minimum, but will sometimes converge to a false pattern (wrong local minimum) rather than the stored pattern (expected local minimum).

A Hopfield network has various units, which have a binary state (1/0). The units update asynchronously or synchronously with the following rule:

Here, $S_{i}$ is the i-th unit of the Hopfield network and $\theta_{i}$ is the threshold. One can define an energy term as:

$$E = - \frac{1}{2}\sum_{i,j}^{}w_{ij}S_{i}S_{j} + \sum_{i}^{}\theta_{i}S_{i}$$

With each update step, the energy either stays constant or decreases.

Reservoir Computing

Reservoir computing is a framework for computation that may be viewed as an extension of neural networks. Typically an input signal is fed into a fixed (random) dynamical system called a reservoir (for example an RNN). Hereby, the dynamics of the reservoir map the input to a higher dimension. Then, a simple readout mechanism is trained to read the state of the reservoir and map it to the desired output. The main benefit is that training is performed only at the readout stage and the reservoir is fixed. A really cool thought is, that basically every (abstract or physical) dynamical system can be used as the reservoir, including a water tank, an electronic circuit or parts of the brain itself.

• Echo State Network: Recurrent neural network with a random and sparsely connected (1%) hidden layer / reservoir, works in discrete time, different activity update modes.
• Liquid State Machine: Biologically plausible, spiking RNN reservoir, randomly connected, continuous in time, asynchronous integration, uses linear discriminate units.

Where is the memory? In the dynamic traces of activity.

Learning Dynamics/Algorithms

• Learning via Backpropagation Through Time (BPTT)
• Learning via Real Time Recurrent Learning (RTRL)
• Reservoir Computing: reservoir capture features of the dynamics can be used to generate them.
• FORCE: First-Order, Reduced and Controlled Error (FORCE) learning. In all three cases shown in the following figure, a recurrent generator network with firing rates r drives a linear readout unit with output z through weights w (red) that are modified during training. Only connections shown in red are subject to modification.

(A) Feedback to the generator network (large network circle) is provided by the readout unit. (B) Feedback to the generator is provided by a separate feedback network (smaller network circle). Neurons of the feedback network are recurrently connected and receive input from the generator network through synapses, which are modified during training. (C) A network with no external feedback. Instead, feedback is generated within the network and modified by applying FORCE learning to the synapses internal to the network.

Self-Organizing Recurrent Networks (SORN)

It combines three distinct forms of local plasticity to learn spatio-temporal patterns in its input while maintaining its dynamics in a healthy regime suitable for learning. The SORN learns to encode information in the form of trajectories through its high-dimensional state space reminiscent of recent biological finding on cortical coding. All three forms of plasticity are shown to be essential for the network's success.

1. STDP rule.
2. Weight Normalization Rule.
3. Distributed Processing Rule.

Process:

1. Start with random network.
2. Add biological-like learning rules.
3. Train readout layer linearly.

Long-Short-Term Memory (LSTM) Networks

Long-Short-Term Memory (LSTM) Networks Long-Short-Term Memory (LSTM) is a feature of a RNN that tackles the problems arising from long sequences / deep networks by a clever memory management. A common LSTM unit is composed of a cell, an input gate i (whether to write to cell), an output gate o (how much to reveal cell), a forget gate f (whether to erase cell) and a gate gate g (how much to write cell). The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell. A comparison between a normal RNN cell and a LSTM cell is given in the figure (a comparison between a normal RNN cell A and a LSTM cell B).

The gate vector can be written as:

where $W = \left( W_{i},W_{f},W_{o},W_{g} \right)$. The cell state is defined as the following:

And the hidden state is a function of the cell state:

The practicality of having this particular cell structure is evident if we look at multiple cells at once, i.e., the processing over multiple sequences, as it is shown in the following figure. Training works again with Back-Propagation-Through-Time. The gradient can now be passed without being interrupted, i.e., the problems of costly weight updates, vanishing and exploding gradients should not occur anymore.

In the figure: Illustration of LSTM over many sequences. Red arrow denotes the gradient, which can flow uninterruptedly.

Biological Plausibility

• BackPropagationn Through Time
  • The brain does not perform BPTT that we know of.
  • But BPTT can be used to train networks that behave like the brain.
• Is Reservoir Computing Better?
  • No one things the brain is as simple as RC...
  • But useful simple model.

Across-Layer Recurrence for Learning Multiple implementation of learning algorithms (usually backprop through time) require feedback from higher layers.

• Through symmetric connections.
• By differentiating inputs and errors.
• Reusing the Predictive Coding setting.
• Many more...

Challenges

• Temporal Credit Assignment in Biological RNNs.
• Structural Priors for Better Computation.
• Bottom-up vs. Top-down Information.
• Integrating Multiple Timescales.
• Finding and Understanding Learning Rules.

Recap

• The Brain uses recurrent connections
  • Anatomically, recurrence is widely visible
  • Functionally, recurrent models capture neural activity better.
• Backpropagation Through Time is difficult
  • Vanishing/Exploding gradients, and temporal credit assignment.
• RNNs keep memory in their dynamics
  • Relationship memory-stability and edge of chaos.
• Biological learning rules can be used in RNNs
  • Hopfield Networks learn associations through Hebbian Learning
  • SORN learns sequences through STDP + Homeostasis.
• Recurrent btw layers can improve inference
  • Predictive Coding or by layer unrolling.
• Higher brain areas are more recurrent
  • Evidence: takes longer to process challenging tasks.
• LSTMs work as RNNs
  • Bypass the backprop problems with specialized memory units
  • More interpretable than standard RNNs.
• Recurrence across layers can be leveraged for learning.