Notes

Content of the Lecture

• Recap: Spiking Neuron Models
• Aim: Supervised Learning in Multi-Layer Spiking Networks (Sequence-to-Sequence task)
• Problems:
  • Spikes Ill-defined Derivative
  • Temporal Credit Assignment
• Solutions:
  • Brief Review Existing Smoothing Solutions.
  • Derivation of Bio-Plausible Surrogate Gradients Learning Rule: SuperSpike.
• Explore Capabilities of Spiking Networks with Fewer Approximations.
• Take a Brief Journey Through a Spiking Networks Optimization Landscape.
• Look at Sensitivity to Surrogate Nonlinearity,
• Approach: BackPropagation Through Time & Surrogate Gradients (Classification Tasks).

Spiking Neuron Models

Motivation Neurobiology mostly uses spiking neural networks. Neurons output spikes, which are binary events and localized in time. So how do hidden units learn?

Bottom-Up Approach:

• Start with a random network model.
• Include data driven plasticity model.
• Observe function Limited success in learning useful hidden layer representations.

One outcome would be Spike-Time Dependent Plasticity (STDP), which is a way the weights can be adjusted. So far, not very useful in building networks; the weights tend to blow up. Over the years, people went over this concept and tried to improve.

Top-Down Approach:

• Start with a function in mind.
• Derive suitable plasticity rules.
• Build functional network models.

Deep Learning is an example of a top-down framework. Two questions remain:

• The Algorithmic Question: How to compute the gradient?
• The Conceptual Question: Which functions are learned?

Recap: Spiking Neuron Models

• Spiking Networks consist of Spiking Neurons.
• Network Modelling largely relies on Simplified Neuron Models.

Biophysics of Neuronal Signal Transmission

From Biophysical to Reduced Neuron Models In order to build a phenomenological model of neuronal dynamics, we describe the critical voltage for spike initiation by a formal threshold $\theta$. If the voltage $U_{i}(t)$ (that contains the summed effect of all inputs) reaches $\theta$ from below, we say that neuron i fires a spike. The moment of threshold crossing defines the firing rate $t_{i}^{(f)}$. The models makes use of the fact that neuronal action potentials of a given neuron always have roughly the same form. If the shape of an action potential is always the same, then the shape cannot be used to transmit information: rather information is contained in the presence or absence of a spike. Therefore action potentials are reduced to "events" that happen at a precise moment in time.

Leaky Integrate-and-Fire Neuron Neuron models where action potentials are described as events are called "Integrate-and-Fire" models. No attempt is made to describe the shape of an action potential. Integrate-and-Fire models have two separate components that are both necessary to define their dynamics:

1. An equation that describes the evolution of the membrane potential $U_{i}(t)$.
2. A mechanism to generate spikes.

The variable $U_{i}$ describes the momentary value of the membrane potential of neuron i. In the absence of any input, the potential is at its resting value $U_{rest}$. If an experimentalist injects a current $I(t)$ into the neuron, or if the neuron receives synaptic input from other neurons, the potential $U_{i}(t)$ will be deflected from its resting value. The basic electrical circuit representing a leaky integrate-and-fire model consists of a capacitor C in parallel with a resistor R driven by a current $I(t)$, as shown in the figure below. The differential equation for describing the leaky-integration of the voltage is given by:

where $\tau = RC$ is the time constant of the circuit.

Now, the second part of the leaky integrate-and-fire neuron is the firing and re-setting of the voltage after the neuron-specific threshold has been reached. At the firing time: $t^{f}:U\left( t^{f} \right) = \ \theta$, the neuron fires (with a not-here-to-be-defined spike-form), the firing time is noted and immediately after the voltage reset to a new value $U_{rest} < \ \theta$:

Exponential Postsynaptic Currents

We also want to model the synapse. Activation of a presynaptic neuron results in a release of neurotransmitters into the synaptic cleft. The transmitter molecules diffuse to the other side of the cleft and activate receptors that are located in the postsynaptic membrane. In both cases, the activation of the receptor results in the opening of certain ion channels and, thus, in an excitatory or inhibitory postsynaptic transmembrane current (EPSC or IPSC). The main mechanism for carrier transport underlying this current is diffusion of the ions passing from the extracellular space into the cell. Instead of developing a mathematical model of the transmitter concentration in the synaptic cleft, we keep things simple and describe transmitter-activated ion channels as an explicitly time-dependent conductivity. This conductivity change most often modelled as an exponentially decaying unction, to represent the effect of closing ion channels. The following differential equation describes the evolution of the postsynaptic current $I(t)$:

Considering the changes arising due to the discrete APs, the term S(t), added in the equation, determines an instantaneous increase in the postsynaptic current proportional to the synaptic weight.

The Spike Response Model (SRM0) So far, we have described neuronal dynamics in terms of systems of differential equations. There is another approach called the "filter picture". In this picture, the parameters of the model are replaced by (parametric) functions of time, generically called "filters". The neuron model is therefore interpreted in terms of a membrane filter as well as a function describing the shape of the spike and, potentially, also a function for the time course of the threshold. Together, these three functions establish the Spike Response Model (SRM). Mathematically speaking, we integrate over the differential equation, then replace the integration times multiplications with convolutions of filter kernels over the spikes:

Supervised Learning in Multi-Layer Spiking Networks - Introduction

Towards Functional Neural Network Models We want this:

Dealing with the Vanishing Gradient Problem Defining the Problem Can we do supervised learning in spiking multi-layer networks with a local online learning rule? We want to compare the output spikes with the target spikes. Let's try:

Van Rossum Distance between Output and Target Spike Trains There are different ways of representing a spike train. If the spikes are seen to be discrete units, the spike train S(t) is given simply by:

Replacing the delta function associated with each spike with an exponential function, that is, add an exponential tail to all spikes, leads to another definition of a spike train:

where H(t) is the heaviside function. The loss between the target spikes distance $\widehat{S}$ and the output spikes distance S can be defined as the Van Rossum distance:

The problem with our spike model and such a loss becomes evident when we try to differentiate the loss with respect to the single weights:

The second partial derivative is problematic because for most neuron models, it is zero except at spike times at which it is not defined. Thus, it forces the gradient to vanish.

A History of Struggle

• Noise Injection
• Differentiate Firing Times
• Make Spikes Differentiable
• Force Hidden Units "On Target"
• Firing-Rate Approaches

Surrogate Gradients & SuperSpike Idea: Replace the non-differentiable Heaviside function with the differentiable sigmoid function $\sigma$, but only in the backward-pass. In the forward pass, leave it as a Heaviside function. The equivalent in machine learning would be "Straight-through estimators". This procedure leads to the replacements:

If now the membrane potential $U_{i}(t)$ is written in the integral form as a spike response model (SRM0)

where $\epsilon$ is the causal membrane kernel (corresponding to the postsynaptic potential) and $\eta$ captures spike dynamics and reset. With some steps that are briefly explained in the paper, one gets for the gradient descent learning rule for a single neuron the following expression:

Here, r is the learning rate, $e_{i}(s) = \lambda*\left( \widehat{S_{i}} - S_{i} \right)$ the error signal and $\lambda$ the eligibility trace ("Ca transient"). This learning rule is called SuperSpike. We can divide this rule into three factors:

• Pre: Presynaptic Activity
• Post: Postsynaptic Activity
• Error Signal: Specific Feedback

The pre- and postsynaptic activity are combined in a multiplicative manner, which can be seen as the Hebbian term (which is "STDP"-like). $\sigma'$ is the voltage nonlinearity, thus the learning rule is voltage based.

Hidden Layers What about training the hidden layers? The learning rule for hidden weights is:

Biologically seen this is problematic, because:

1. It requires symmetric weights
2. There are downstream activities

One way to overcome those issues is by applying feedback-alignment. Not that all quantities computed online. Temporal credit assignment through dynamics at the synaptic level (eligibility trace).

In the figure: Network trained to solve a non-linearly separable classification problem with noisy input neurons. (a) Sketch of network layout with two output units and four hidden units. (b) Snapshot of network activity at the end of training with random feedback. Four input patterns from two non-linearly separable classes are presented in random order 8shaded areas). In between stimulus periods, input neurons spike randomly with 4Hz background firing rate. (c) Learning curves of 20 trials with different random initializations (gray) for a network with random feedback connections that solves the task. The average of all trials is given by the black line. The average of 20 simulation trials with an additional regularization term is shown in green. (d) Same as panel c but for symmetric feedback. (f) Same as panel c but for uniform ("all ones") feedback connections.

Supervised Learning in Multi-Layer Spiking Networks - Seq2Seq Learning

Sequence-To-Sequence Learning Because zero error was achieved with simple tasks using different types of feedback signal, the learning rule can be put to work in harsher conditions, under more challenging tasks: making it associate a spatio-temporal target output pattern to a repeating frozen Poisson noise input. In this case, a larger, 3-layer net was used (100 in, variable number hidden, 100 out), but an output pattern matching the target was achieved with only 32 hidden neurons. The random feedback performs worse than a network that was trained without a hidden layer, but with symmetrical weights.

What about Unsupervised Learning? A Spiking Auto-Encoder with "Gaussian" Input

The rule can also be used as an auto-encoder network, since it can be provided the same pattern as both input and output. It is able to reconstruct the output pattern with high fidelity while having a number of hidden units smaller than the number of units in the input and output layers.

Spiking Nets and Temporal Coding The RNN term is used in its widest sense, that of networks with states evolving in time based on well/defined dynamic recurrent equations. An important fact to note is that, while recurrent synaptic connections between neurons in a network give rise to recurrent dynamics, they are not absolutely necessary, as dynamical recurrent can arise without them. This is the case of neurons or synapses which have state which evolve according to internal dynamics: the current state depends on the previous state and the next state depends on the current state, thus state-full units are inherently recurrent. This idea can be very well applied to SNNs, with computations necessary to update a cell state that can be unrolled in time as seen in the figure below.

In the figure: Illustration of the computational graph of a SNN in discrete time. Time steps flow from left to right. Input spikes S(0) are fed into the network from the bottom and propagate upwards to higher layers. The synaptic currents I are decayed by $\alpha$ in each time step and fed into the membrane potentials U. The U are similarly decaying over time as characterized by $\beta$. Spike trains S are generated by applying a threshold non-linearity to the membrane potentials U in each time step. Spikes causally affect the network state (orange connections). First, each spike causes the membrane potential of the neuron that emits the spike to be reset. Second, each spike may be communicated to the same neuronal population via recurrent connections V(1). Finally, it may also be communicated via W(2) to another downstream network layer or, alternatively, a readout layer on which a cost function is defined.

The Backpropagation rule can be applied to RNNs. In this case the recurrence is "unrolled" meaning that an auxiliary network is created by making copies of the network for each time step. The unrolled network is simply a deep network with shared feed-forward weights W(1) and recurrent weights V(1), on which the standard BP applies:

Applying BP to an unrolled network is referred to as Back-Propagation Through Time (BPTT).

Neuromorphic Hardware & Spiking Neural Networks

Neuromorphic Engineers Build Hardware that Seeks to Emulate Neural Networks Instead of Simulating Them To make things more efficient, we would like to create an hardware that does most of the simulation through physical properties rather than simulating them through software. One major issues of these hardware is related to Device Mismatch (a stumbling block for widespread use of analog neuromorphic hardware), i.e., a slight difference in membrane potential among chips due to manufacturing variability, which is not found in software simulations. So, can analog neuromorphic substrates self-calibrate through surrogate gradient learning and overcome device mismatch? To study this question we used BrainScaleS-2 analog neuromorphic hardware system and if you record from a neuron in this chip when a current is injected you can capture the analog voltage through an oscilloscope.

In-the-loop Surrogate Gradient Training Forward-pass on chip and backward pass in software.

1. Forward pass on chip.
2. Measure on-chip analog voltage traces.
3. Inject true voltages into comp. graph.
4. Compute surrogate gradients update weights.

Functional spiking neural networks trained on accelerated analog neuromorphic hardware. We show that on the MNIST example training loss goes basically to zero. Surrogate gradient learning self-calibrates the analog neuromorphic substrate.

• Surrogate gradients with BPTT are a powerful tool to build functional spiking neural networks.
• Can instantiate sparsely active networks.
• Voltage-aware in-the-loop training with surrogate gradients self-calibrates neuromorphic hardware.