Notes

Towards Functional Neural Network Models We want this:

Dealing with the Vanishing Gradient Problem Defining the Problem Can we do supervised learning in spiking multi-layer networks with a local online learning rule? We want to compare the output spikes with the target spikes. Let's try:

Van Rossum Distance between Output and Target Spike Trains There are different ways of representing a spike train. If the spikes are seen to be discrete units, the spike train S(t) is given simply by:

Replacing the delta function associated with each spike with an exponential function, that is, add an exponential tail to all spikes, leads to another definition of a spike train:

where H(t) is the heaviside function. The loss between the target spikes distance $\widehat{S}$ and the output spikes distance S can be defined as the Van Rossum distance:

The problem with our spike model and such a loss becomes evident when we try to differentiate the loss with respect to the single weights:

The second partial derivative is problematic because for most neuron models, it is zero except at spike times at which it is not defined. Thus, it forces the gradient to vanish.

A History of Struggle

• Noise Injection
• Differentiate Firing Times
• Make Spikes Differentiable
• Force Hidden Units "On Target"
• Firing-Rate Approaches

Surrogate Gradients & SuperSpike Idea: Replace the non-differentiable Heaviside function with the differentiable sigmoid function $\sigma$, but only in the backward-pass. In the forward pass, leave it as a Heaviside function. The equivalent in machine learning would be "Straight-through estimators". This procedure leads to the replacements:

If now the membrane potential $U_{i}(t)$ is written in the integral form as a spike response model (SRM0)

where $\epsilon$ is the causal membrane kernel (corresponding to the postsynaptic potential) and $\eta$ captures spike dynamics and reset. With some steps that are briefly explained in the paper, one gets for the gradient descent learning rule for a single neuron the following expression:

Here, r is the learning rate, $e_{i}(s) = \lambda*\left( \widehat{S_{i}} - S_{i} \right)$ the error signal and $\lambda$ the eligibility trace ("Ca transient"). This learning rule is called SuperSpike. We can divide this rule into three factors:

• Pre: Presynaptic Activity
• Post: Postsynaptic Activity
• Error Signal: Specific Feedback

The pre- and postsynaptic activity are combined in a multiplicative manner, which can be seen as the Hebbian term (which is "STDP"-like). $\sigma'$ is the voltage nonlinearity, thus the learning rule is voltage based.

Hidden Layers What about training the hidden layers? The learning rule for hidden weights is:

Biologically seen this is problematic, because:

1. It requires symmetric weights
2. There are downstream activities

One way to overcome those issues is by applying feedback-alignment. Not that all quantities computed online. Temporal credit assignment through dynamics at the synaptic level (eligibility trace).

In the figure: Network trained to solve a non-linearly separable classification problem with noisy input neurons. (a) Sketch of network layout with two output units and four hidden units. (b) Snapshot of network activity at the end of training with random feedback. Four input patterns from two non-linearly separable classes are presented in random order 8shaded areas). In between stimulus periods, input neurons spike randomly with 4Hz background firing rate. (c) Learning curves of 20 trials with different random initializations (gray) for a network with random feedback connections that solves the task. The average of all trials is given by the black line. The average of 20 simulation trials with an additional regularization term is shown in green. (d) Same as panel c but for symmetric feedback. (f) Same as panel c but for uniform ("all ones") feedback connections.