Notes

Motivation: Why Is It So Important to Understand RNN Learning?

• The mammalian cortex is highly recurrent - it will help us to better understand the brain.
• The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: sequences in the input, the output, or in the most general case both. Thus, better understanding RNNs might help us to develop new, powerful RNN algorithms that are able to learn long and complex sequences.
• In programming terms be interpreted as running a fixed program with certain inputs and some internal variables. Viewed this way, RNNs essentially describe programs. In fact, it is known that RNNs are Turing-Complete in the sense that they can be used to simulate arbitrary programs (with proper weights).
• RNNs perform exceptionally well in language-modelling, the task of predicting the probability of the next word in a sequence.

RNNs in the Brain There are claims that networks of the AlexNet type successfully predict properties of neurons in visual cortex. Thus, one natural question arises: how similar is an ultra-deep residual network to the primate cortex? A notable difference is the depth. While a residual network has many as 1202 layers, biological systems seem to have two orders of magnitude less, if we make the customary assumption that a layer in the NN architecture corresponds to a cortical area. In fact, there are about half a dozen areas in the ventral stream of visual cortex from the retina to the Inferior Temporal Cortex. Notice that it takes in the order of 10ms for neural activity to propagate from one area to another one (remember that spiking activity of cortical neurons is usually well below 100Hz). The evolutionary advantage of having fewer layers is apparent: it supports rapid (100ms from image onset to meaningful information in IT neural population) visual recognition, which is a key ability of human and non-human primates. It is intriguingly possible to account for this discrepancy by taking into account recurrent connections within each visual area. Areas in visual cortex comprise six different layers with lateral and feedback connections, which are believed to mediate some attentional effects and even learning (such as backpropagation). "Unrolling" in time the recurrent computations carried out by the visual cortex provides an equivalent "ultra-deep" feedforward network, which might represent a more appropriate comparison with the state-of-the-art computer vision models.

Recurrent Projections in the Cat Brain Paper: "A Quantitative Map of the Circuit of Cat Primary Visual Cortex".

By mapping the circuit of cat primary visual cortex (V1), it is evident, that there are recurrent projections involved.

Anatomical Evidence for RNNs in the Rodent Brain Paper: "Distinct Timescales of Population Coding Across Cortex".

(Train a mice to turn left or right depending on sound location and Record neural activity in Auditory Cortex and Posterior Parietal Cortex). Similarly, it has been shown in rodents, that the communication between columns is organized by multiple highly specific horizontal projection patterns. Population coding is a method to represent stimuli by using the joint activities of a number of neurons. In population coding, each neuron has a distribution of responses over some set of inputs, and the responses of many neurons may be combined to determine some value about the inputs.

Anatomical Evidence for RNNs in the Primate Brain The brain has both a feed-forward structure and recurrent pathways. Information can get sent back from one area to a previous one or echo around the same area multiple times. Studies suggest this extra processing helps the brain interpret challenging visual information, such as objects that are occluded or viewed from unusual angles. A recent study found images that are difficult for a feed-forward model to classify but easy for humans and monkeys to interpret, although they take slightly longer to classify these challenging images than normal ones. This delay suggests that some recurrent processing is involved. The researchers then looked at how neural activity in the monkey's brain evolves as these images are processed. A benefit of convolutional neural networks is that the response of different layers in the model can be used to predict the response of neurons in different brain areas. The researchers found that the feed-forward model predicts the activity of neurons fairly well at early stages (up to 0.1s into the response) but struggles at later time points. When a convolutional neural network is not performing well, researchers in computer vision tend to add more layers to it, making it "deeper". The authors tested whether such deeper networks could better predict neural responses to their challenging images, under the assumption that a network with more layers, which computes over space, resemble recurrent pathways, which compute over time. These deeper networks were indeed better than the shallower model at predicting neural activity at later time points. Finally, the authors added recurrent connections to the structure of their original model and found that responses at later time points in the model better matched later time points in the data. Specifically, when recurrent connections were added to this "shallower" network, it predicted neural activity as well as the deeper model did. Overall, this work strongly suggests that recurrent processing is an important contributor to computation in the visual system.

In the image below: Both primates and feedforward DCNNs were tasked to identify which object is present in each test image (1320 images). Top: the stages in the primate ventral visual pathway (retina, LGN, V1, V2, V4, and the IT cortex), which is implicated in core object recognition. We can conceptualize each stage as rapidly transforming the representation of the image and ultimately yielding the primates' behavior (i.e., producing a behavioral report of which object was present). The blue arrows indicate the known anatomical feedforward projections from one area to the other. The red arrows indicate the known lateral and top-down recurrent connections. Bottom: a schematic of a similar pathway commonly present in DCNNs. These networks contain a series of convolutional and pooling layers with nonlinear transforms at each stage, followed by fully connected layers (which approximate macaque IT neural responses) that ultimately gives rise to the models' "behavior". Note that the DCNNs only have feedforward (blue) connections.

Functional Evidence Generate two models of neural activity incorporating any variable we can think of with a Generalized Linear Model (GLM). The predictors can be trained in isolation (uncoupled) or dependent on previous neuron activity (coupled). The Coupled model performs much better for PPC, ergo we assume the recurrence is important. For AC both perform similarly, but AC is less recurrent than PPC.