Notes

Lecturer: Giacomo Indiveri

Content of the Lecture

• Neuromorphic Engineering
• Neuromorphic Electronic Circuits
  • The Differential Pair Circuit
  • Silicon Neurons
  • Differential Pair Integrator Circuit
• Neuromorphic Processors
  • Learning
  • Computational Primitives
• Conclusions

VLSI

• Very Large Scale Integration Technology allows to fabricate chips and memory.
• VLSI are usually digital, high power, not fault tolerant or robust, and clocked (synchronous), not massively parallel. The failure of one transistor is the failure of the computer.

The computer hardware had a radical paradigm shift when we looked to real brains. For instance, a bee brain is much smaller and consumes much less power (using neurons in a slow way), and offer real time interaction with the environment and complex behavior.

The Term "Neuromorphic" The term neuromorphic was coined by Carver Mead in the late '80s to describe VLSI systems containing electronic analog/digital circuits that exploit the physics of silicon to reproduce the bio-physics of neural circuits present in the nervous system. It is a discipline characterized by two main goals.

• To understand the computational properties of biological neural systems using standard CMOS VLSI technology as a tool.
• To exploit the known properties of biological systems to design and implement efficient devices for engineering applications.

New hardware different from conventional computers: radically different from Von Neumann architectures. Now, there are parallel elements with memory and computation co-localized, with continuous streaming data driven computation, no clock. The co-localization of memory and computation allows to have no I/O bottleneck and no memory bottleneck.

The INI Neuromorphic Engineering Mission Learn to build artificial neural processing systems that can interact intelligently with the physical world.

• Combine multiple disciplines (neuroscience, physics, computer science, electrical engineering, ...).
• Exploit device physics to directly emulate the biophysics of neural systems.
• Let time represent itself.
• Implement robust computation in autonomous agents that produce cognitive behavior.

Neuromorphic Computing vs Engineering Neuromorphic computing uses a dedicated VLSI hardware, high performance computing, it is application driven and uses conservative approaches. Neuromorphic engineering is a fundamental research, deeply rooted in biology. It emulates neural function in subthreshold analog and asynchronous digital.

Neuromorphic Electronic Circuits

• Reproduce physics of neural computation using subthreshold analog circuits and asynchronous digital circuits.
• Build autonomous learning behaving systems that can interact with the environment in real-time.
• Best exploit for current and future VLSI technologies.
• Suited for nano and emerging technologies.
• Ideal tools for real- and accelerated-time modeling of neural systems.
• Compact, low-power sensory processing devices.
• Can interface directly with living systems.

Circuits Digital transistors operating only in the minimum and the maximum. Analog transistors use also intermediate amounts, thus transistors can emulate physical proteic channels. In biology, at high voltages, the fraction of the channels that are open approaches unity, causing a saturation. The same can be seen in a subthreshold regime. In subthreshold, the current is smaller than 1V, it increases exponentially, and after threshold currents change quadratically. It changes from pico to nano amps.

In Complementary Metal-Oxide Semiconductor (CMOS) technology, there are two types of MOS-FETs: n-FETs and p-FETs. There is no current going to transistors. In traditional CMOS circuits, all n-FETs have the common bulk potential $V_{b}$ connected to ground (GND) and all p-FETs have a common bulk potential connected to the power supply rail $\left( V_{dd} \right)$.

The current is defined to be positive if it flows from the drain to the source.

Diffusion and Saturation

Output Current versus $V_{gs}$ and $V_{ds}$ and Current Source

The Current Mirror

The output current is a mirrored copy of the input current. If both MOSFETs are of the same size and have the same source voltage, they source the same current, which is why the device is called current mirror. The input current $I_{in}$ through the diode-connected transistor $M_1$ sets the common gate voltage $V_g$ and hence the output current $I_{out}$ of the second transistor $M_2$.

The output current can be scaled by choosing different transistor sizes, or by choosing different source potentials $V_{s1}$ and $V_{s2}$ for the two MOSFETs. If $M_2$ is in saturation:

$$I_{out} = e^{(V_{s1} - V_{s2})/U_{T}}I_{in}$$

The Differential - Pair

Note that in $I_2$ equation $V_1$ at the numerator is a mistake, should instead be $V_2$.

To implement the difference of currents ($I_1$ - $I_2$) we can use the current-mirror circuit.

The Transconductance Amplifier For small differential voltages (e.g., $\left| V_{1} - V_{2} \right| < 200mV$), the tanh() relationship is approximately linear and the equation $I_{out} = I_{b}\tanh\left( \frac{k}{2U_{T}}\left( V_{1} - V_{2} \right) \right)$ can be reduced to: $I_{out} \approx g_{m}\left( V_{1} - V_{2} \right)$ where $g_{m} = \frac{I_{b}k}{2U_{T}}$.

Spike Generating Mechanism If the membrane voltage increases above a certain threshold, a spike-generating mechanism is activated and an action potential is initiated.

A Conductance-Based Silicon Neuron In 1991 Misha Mahowald and Rodney Douglas proposed a conductance-based silicon neuron and showed that it had properties remarkably similar to those of real cortical neurons.

Neuron Models Traditionally there have been two main classes of neuron models:

• Conductance-Based (R-C)
• Integrate and Fire (I-C) But recently proposed models bridge the gap between the two:
• Generalized Integrate and Fire This model can account for a very large set of behaviors captured by far more complicated Hodgkin-Huxley models. $\frac{d}{dt}u_{mem} = \frac{i_{in}}{C_{mem}} + F(u_{mem})$ where $F(u_{mem})$ is a non-linear function of $u_{mem}(t)$.

Current-mode CMOS circuits operated in the subthreshold, or weak-inversion regime can be used to implement log-domain filters. An example of classical log-domain integrator is presented in the picture below. This circuit's linear transfer function can be easily derived by applying the translinear principle on the $V_{gs}$ loop highlighted by the arrows: given the exponential relationship between the subthreshold currents of the p-FETs and their $V_{gs}$ voltages, we can write: $I_{th} \bullet I_{1} = I_{2} \bullet I_{out}$. In subthreshold, the output n-FET $M_{out}$ produces a current that changes exponentially with its gate voltage $V_{c}$. Differentiating $I_{out}$ with respect to $V_{c}$ and combining the result with the capacitor equation $C\frac{d}{dt}V_{C} = I_{2} - I_{\tau}$ we obtain: $\tau\frac{d}{dt}I_{out} + I_{out} = \frac{I_{th}}{I_{\tau}}I_{in}$.

The DPI is a CMOS current-mode circuit that operates in the subthreshold regime integrating voltage pulses. However, rather than using a single p-FET to generate the appropriate $I_{w}$ current, via the triangular principle (Gilbert, 1975), it uses a differential pair in negative feedback configuration. This allows the circuit to achieve LPF functionality with tunable dynamic conductances: Input voltage pulses are integrated to produce an output current that has maximum amplitude set by $V_{w}$, $V_{t}$ and $V_{thr}$. (Silicon neuron circuits) It has additional advantages of providing a compact layout, better matching properties and lower power consumption. The differential - pair integrator is used to model synaptic dynamics. It comprises only 3 n-FETs, 2 p-FETs and 1 capacitor. The two current sources are implemented using two subthreshold MOSFETs: one n-FET for the $I_{in}$ current and on p-FET for the $I_{t}$ current. Following a similar derivation to the one used in the classical log-domain integrator, the characteristic equation is obtained as observed in the picture below.

Additional circuits can be attached to the DPI synapse to extend the model with extra features typical of biological synapses and implement various types of plasticity. For example, by adding two extra transistors, we can implement voltage-gated channels that model NMDA synapse behavior. Similarly, by using two more transistors, we can extend the synaptic model to be conductance based. Furthermore, the DPI circuit is compatible with previously proposed circuits for implementing synaptic plasticity, on both short timescales with models of short-term depression (STD) and on larger timescales with spike-based learning mechanisms, such as spike timing-dependent plasticity (STDP). The DPI neuron circuit is a variant of the generalized IF neuron and is depicted in the following picture. The input DPI low-pass filter (yellow, ML1 - ML3) models the neuron's leak conductance. A spike event generation amplifier (red, MA1 - MA6) implements current-based positive feedback (modeling both sodium activation and inactivation conductances) and produces address-events at extremely low-power. The reset block (blue, MR1 - MR6) resets the neuron and keeps it in a reset state for a refractory period, set by the $V_{ref}$ bias voltage. An additional DPI filter integrates the spikes and produces a slow after hyper-polarizing current $I_{g}$ responsible for spike-frequency adaptation (green, MG1 - MG6). By applying a current-mode analysis to both the input and the spike-frequency adaptation DPI circuits, it is possible to derive a simplified analytical solution:

$$\tau\frac{d}{dt}I_{mem} + I_{mem} \approx \frac{I_{th}I_{in}}{I_{\tau}} - I_{g} + f\left( I_{mem} \right)$$

$$\tau_{ahp}\frac{d}{dt}I_{g} + I_{g} = \frac{I_{thr}I_{ahp}}{I_{\tau_{ahp}}}$$

The state of the art version of this neuron circuit consumes one order of magnitude less power than the circuit described in the following figure and two orders of magnitude less power than the digital implementation of the I&F neuron. Given the exponential nature of the generalized IF neuro's non-linear term $f\left( I_{mem} \right)$, the DPI-neuron implements an adaptive exponential IF model. This IF model has been shown to be able to reproduce a wide range of spiking behaviors, and explain a wide set of experimental measurements from pyramidal neurons.

Neuromorphic Processors Typical spiking neural network chips have the elements described in the figure below. Multiple instances of these elements can be integrated onto single chips and connected among each other either with on-chip hard-wired connections or via off-chip reconfigurable connectivity infrastructures. The most relevant characteristics of processors build based on analog circuits working in subthreshold are:

• Slow temporal, non-linear dynamics
• Massively parallel operation
• Inhomogeneous, imprecise and noisy
• Adaptation and learning is done at multiple time-scales
• Fault-tolerant and mismatch insensitive by design
• Fast asynchronous digital routing circuits
• Reprogrammable network topology and connectivity.

Neuromorphic vs Conventional Processors

Winner - Take - All Networks in Neuromorphic Hardware "Winner take all" (WTA) refers to a type of neural network architecture that is commonly used in neuromorphic hardware. In a WTA network, each neuron competes with other neurons to be the "winner" and output the highest value. This can be useful in situations where you want to identify the most active or strongest signal among a group of neurons. This architecture is inspired by the way the brain works where many neurons compete to fire.

Learning and Winner - Take - All Networks Memories can be formed in neuromorphic hardware using attractor networks, which are a type of recurrent neural network. An attractor network is composed of a group of neurons that are connected to each other through synapses. These synapses can be either excitatory, which increases the likelihood that a neuron will fire, or inhibitory, which decreases the likelihood that a neuron will fire. The network's dynamics are determined by the strengths of these connections, which can be modified through a process called synaptic plasticity. When the network is exposed to a specific input pattern, the neurons that are active will strengthen their connections to other active neurons, and inhibitory connections will form between active neurons and inactive neurons.This process creates a stable state, or an attractor, in the network. The attractor corresponds to the input pattern that was presented to the network, and the network will continue to settle to this attractor state even after the input pattern is removed. This behavior allows the network to "remember" the input pattern, and this process is called memory formation. The network can then be used to recall the stored information by providing a partial or noisy version of the input pattern.

Mechanisms operating at the network level can allow neural processing systems to form short-term memories, consolidate long-term ones, and carry out nonlinear processing functions such as selective amplification (e.g., to implement attention and decision making). An example of such a network-level mechanism is provided by "attractor networks". These are networks of neurons that are recurrently connected via excitatory synapses, and that can settle into stable patterns of firing even after the external stimulus is removed. Different stimuli can elicit different stable patterns, which consist of specific subsets of neurons firing at high rates. Each of the high-firing rate attractor states can represent a different memory. To make an analogy with conventional logic structures, a small attractor network with two stable states would be equivalent to a flip-flop gate in CMOS.

A particularly interesting class of attractor networks is the one of soft winner-take-all (sWTA) neural networks. In these networks, groups of neurons both cooperate and compete with each other. Cooperation takes place between groups of neurons spatially close to each other, while competition is typically achieved through global recurrent patterns of inhibitory connections. When stimulated by external inputs, the neurons excite their neighbors and the ones with the highest response suppress all other neurons to win the competition. Thanks to these competition and cooperation mechanisms, the outputs of individual neurons depend on the activity of the whole network and not just on their individual inputs.

Conclusion

• We study the principles of computation of cortical circuits and validate them on neuromorphic systems that interact intelligently with the environment.
• We exploit progress in technology to develop mixed-signal neuromorphic electronic circuits for emulating neural dynamics and learning in real-time.
• We build analog/digital neural processing systems interfaced to sensors and robotic platforms that can (learn to) produce intelligent behavior.