Notes

Neuromorphic Intelligence - Introduction & Relation to Neuroscience

Artificial vs Neuromorphic Intelligence

• Artificial Neural Networks
  • Simulate abstract brain-inspired computing architectures on digital time-multiplexed computing substrates.
• Neuromorphic Architectures
  • Use the physics of electronic devices and circuits to emulate real neurons. They use physical (real) time and circuit dynamics to compute through their time evolution.
  • The physical hardware substrate is the algorithm.

Synapses vs Neurons

• In Biology
  • Pyramidal neurons have thousands of synapses, mainly distributed on the distal dentric arborization.
  • The excitatory drive from 10-40 inputs, discharging at an average rate of 100 spikes/s, should cause the postsynaptic neuron to discharge near 100 spikes/s.
• In Neurons
  • To first order approximation, a neuron has a gain of less than 1/10.
  • To produce one output spike, at least 10 input spikes should arrive in parallel to 10 different synapses.
  • An input spike train arriving onto a single synapse at 1kHz should produce one output spike train at 100Hz.

Dale's Principle In 1935, Sir Henry Dale hypothesized that a neuron extends its metabolic activity from the soma to all of its processes. In 1954, Sir John Eccles reformulated this principle to say that "neurons release the same set of transmitters at all of their synapses".

Dale's Law: In practice Dale's law requires neurons to be only excitatory or only inhibitory. A neuron cannot excite some targets and inhibit others. All projections of excitatory neurons can have only positive synaptic weights. All projections of inhibitory neurons can only have negative weights.

Dendrites The active electrical properties of dendrites shape neuronal input and output and are fundamental to brain function. They can implement analog signal processing, digital logic, and state dependent computations. These properties enable neocortical pyramidal neurons to classify linearly non-separable inputs - a computation conventionally thought to require multilayered networks. Recently it has even been hypothesized that there are multiple "Dendritic solutions to the credit assignment problem".

The Neocortex The neocortex is the newest part of the cerebral cortex to evolve. It is a distinguishing feature of mammals. In humans, it is 90% of the cerebral cortex and 76% of the entire brain.

The Canonical Micro-Circuit Across all areas of Neocortex

• Most cortical connections are local and excitatory.
• Most cortical excitatory synapses and almost all inhibitory ones originate from cortical neurons.
• Most of these synapses originate from neurons within the local cortical area.

Cortical Computation

• It is carried out almost entirely by local circuits.
• Recurrent circuits perform signal restoration.
• Control of gain is critical for cortical computation.

Model of Synapses

The First Models of Neurons

• Lapicque: The Integrate and Fire Model

• McCulloch & Pitts Neuron: "A logical calculus of the ideas immanent in nervous activity".

• Hodgkin & Huxley Model
  • Cell Lipid Bilayer (Membrane): Capacitance/Area
  • Leak Channels: Linear Conductance
  • Voltage-Gated Ion Channels: Conductance (V,t), voltage-dependent open-close probability.

Models of Neurons in Neural Networks The McCulloch&Pitts model quickly became extremely popular, and dominated the Artificial Neural Network scene for decades. Why? Isomorphism with calculus of logical propositions. In the hands of John Von Neumann, the McCulloch&Pitts model became the basis for the logical design of digital computers.

Neuromorphic Engineering Approach

The Deep Network Revolution Although the first successes of ANNs were first demonstrated in the 1980's they only started to outperform classical optimization and engineering approaches from 2009 on. In 2011, CNNs trained using backpropagation on GPUs achieved for the first time superhuman performance in a visual pattern recognition contest.

Deep Networks Galore

• As CNNs and DNNs outperformed classical approaches, many research groups started to extend and optimize them.
• The AI field is now (mostly) dominated by attempts to improve accuracy on standard benchmarks, by scaling up network size and parameter count.
• GPT-3 is a network with 175-billion parameters. It's memory size is exceeding 350GB, and training it requires an estimated $12 million.

Problems and Limitations of AI

The Neuromorphic Engineering Approach Methods

• Study fundamental neuroscience, principles of neuro-physiology, neuro-anatomy, theory of computation, electrical engineering, ...
• Exploit the physics of silicon to reproduce the bio-physics of neural systems, using subthreshold analog VLSI circuits.
• Develop distributed multi-core spiking architectures using asynchronous digital VLSI circuits.
• Build real-time autonomous cognitive agents able to carry out behavioral tasks in complex environment.

Neuromorphic Agents When building processing chips, one can be inspired from life-forms that have brain and use them in every-day tasks, like mammals, insects etc. The physical properties of computational elements in brains can also be used as a reference from the computational resource usage standpoint, for example a bee's brain has a weight of 1 mg, a volume of 1 mm3 and manages to squeeze almost 1 million neurons in it. The energy per operation is approximated to 10−15J/spike. Functional principles used by these life-forms can be used in the development of neuromorphic agents. Some of these are:

• Exploiting physical space in the most efficient way possible.
• Let time represent itself.
• Use both analog and digital computing elements.
• Exploit non-linearities and temporal dynamics.
• Leverage noise, variability and stochasticity.
• Minimize wiring (maximize local connectivity).
• Maximize fine grain parallelism (no time-multiplexing).
• Co-localize memory and computation.
• Use physical time (no clocks).
• Match dynamics and time-constants.
• Optimize for processing complex spatio-temporal (dynamic and noisy) signals.
• Re-use computational principles for sensory processing, motor control and cognitive computing.

Brain-Inspired Computing: A Radical Paradigm Shift Exploit Physical Space

• Use parallel arrays of processing elements.
• Maximize fine grain parallelism (no time-multiplexing).
• To co-localize memory and computation.
• Use passive circuits, data-drive, in continuous time.
• Exploit all the properties of transistors and memristors.

Let Time Represent Itself

• For interacting with the environment in real-time.
• To match the circuit time constants to the input signal dynamics.
• For inherently synchronizing with the real-world natural events.
• To process sensory signals efficiently.

Neuromorphic Synapse Analog Circuits

Synapse Analog Circuits The DPI is a CMOS current-mode circuit that operates in the subthreshold regime integrating voltage pulses. However, rather than using a single p-FET to generate the appropriate $I_{w}$ current, via the triangular principle (Gilbert, 1975), it uses a differential pair in negative feedback configuration. This allows the circuit to achieve LPF functionality with tunable dynamic conductances: Input voltage pulses are integrated to produce an output current that has maximum amplitude set by $V_{w}$, $V_{t}$ and $V_{thr}$. (Silicon neuron circuits) It has additional advantages of providing a compact layout, better matching properties and lower power consumption. The differential - pair integrator is used to model synaptic dynamics. It comprises only 3 n-FETs, 2 p-FETs and 1 capacitor. The two current sources are implemented using two subthreshold MOSFETs: one n-FET for the $I_{in}$ current and on p-FET for the $I_{t}$ current. Following a similar derivation to the one used in the classical log-domain integrator, the characteristic equation is obtained as observed in the picture below.

Additional circuits can be attached to the DPI synapse to extend the model with extra features typical of biological synapses and implement various types of plasticity. For example, by adding two extra transistors, we can implement voltage-gated channels that model NMDA synapse behavior. Similarly, by using two more transistors, we can extend the synaptic model to be conductance based. Furthermore, the DPI circuit is compatible with previously proposed circuits for implementing synaptic plasticity, on both short timescales with models of short-term depression (STD) and on larger timescales with spike-based learning mechanisms, such as spike timing-dependent plasticity (STDP). The DPI neuron circuit is a variant of the generalized IF neuron and is depicted in the following picture. The input DPI low-pass filter (yellow, ML1 - ML3) models the neuron's leak conductance. A spike event generation amplifier (red, MA1 - MA6) implements current-based positive feedback (modeling both sodium activation and inactivation conductances) and produces address-events at extremely low-power. The reset block (blue, MR1 - MR6) resets the neuron and keeps it in a reset state for a refractory period, set by the $V_{ref}$ bias voltage. An additional DPI filter integrates the spikes and produces a slow after hyper-polarizing current $I_{g}$ responsible for spike-frequency adaptation (green, MG1 - MG6). By applying a current-mode analysis to both the input and the spike-frequency adaptation DPI circuits, it is possible to derive a simplified analytical solution:

$$\tau\frac{d}{dt}I_{mem} + I_{mem} \approx \frac{I_{th}I_{in}}{I_{\tau}} - I_{g} + f\left( I_{mem} \right)$$

$$\tau_{ahp}\frac{d}{dt}I_{g} + I_{g} = \frac{I_{thr}I_{ahp}}{I_{\tau_{ahp}}}$$

The state of the art version of this neuron circuit consumes one order of magnitude less power than the circuit described in the following figure and two orders of magnitude less power than the digital implementation of the I&F neuron. Given the exponential nature of the generalized IF neuro's non-linear term $f\left( I_{mem} \right)$, the DPI-neuron implements an adaptive exponential IF model. This IF. model has been shown to be able to reproduce a wide range of spiking behaviors, and explain a wide set of experimental measurements from pyramidal neurons.

Neuromorphic Processors

Neuromorphic Processors Typical spiking neural network chips have the elements described in the figure below. Multiple instances of these elements can be integrated onto single chips and connected among each other either with on-chip hard-wired connections or via off-chip reconfigurable connectivity infrastructures. The most relevant characteristics of processors build based on analog circuits working in subthreshold are:

• Slow temporal, non-linear dynamics
• Analog and digital co-design
• On-Chip inference and learning
• Reconfigurable architecture
• Distributed SRAM and TCAM memory cells
• Capacitors for state dynamics
• Massively parallel operation
• Inhomogeneous, imprecise and noisy
• Adaptation and learning is done at multiple time-scales
• Fault-tolerant and mismatch insensitive by design
• Fast asynchronous digital routing circuits
• Reprogrammable network topology and connectivity
• Ideal for integration with binary, non-volatile resistive memory devices
• Ideal for integration with dynamic, volatile/non-volatile memristive devices
• Ideal for integration in 3D VLSI technology.

Neuromorphic Pros & Cons

Why Spikes?

• Background: Signal processing systems that need to process real-world sensory signals and interact with the environment in real-time.
• Requirements:
  • Low power and low-latency.
  • Local processing, multi-core architectures and distributed computing.
  • Robust communication of signals across long distances through noisy channels.
• Solution: Use Spikes! The optimal method that minimizes bandwidth and power consumption for achieving this goal, under these constraints, is pulse-frequency modulation.

Why Analog? Advantages

• Exploit the full potential of emerging memory technologies
  • Control multi-level properties with analog pulse heights.
  • Exploit intrinsic non-linearities.
  • Exploit intrinsic stochasticity.
  • Exploit non-volatility properties.
• Avoid use of digital clock circuitry.
• Avoid large DAC/ADC overhead.
• Minimize power consumption.

PCM-trace Exploit the drift of PCM devices to implement long-lasting eligibility traces. These enable the construction of powerful learning mechanisms for solving complex tasks by bridging the synaptic (ms) and behavioral time-scales (minutes).

Disadvantages Membrane currents measured across 256 neurons, in response to the same inputs differ. How to cope with mismatch? Integrate over space (use populations of neurons) and Integrate over time (use mean rates).

False Myths about Analog Neural Responses

• Neural Responses are Slow (Population firing rates of neurons can reliably encode weak signal changes approx. 50 times faster than individual neurons, populations of noisy neurons have very fast response times).
• Neurons need to fire at high firing rates to achieve high precision (E-I Balanced populations can encode signals with high precision and low rates, sparse neural population activity can represent signals with high accuracy).
• Neurons need to be accurate to propagate precise information across population layers (Neurons need to be noisy to propagate neural activity reliably, to propagate signals using low-firing rates, it is necessary to use inhomogeneous populations of neurons).

Neuromorphic Applications

Neuromorphic Applications

• Extreme Edge Computing
  • Online Anomaly Detection
    • Industrial Predictive Maintenance (PM): PM involves the health monitoring of a degrading system. Vibration patterns yield valuable information about the health state of a running machine. PM is typically applied to large industrial tasks, but could be useful for small appliances and robots as well.
  • Online HFO Detection
    • High-Frequency Oscillations (HFO) are spontaneous EEG events in the frequency range between 80 and 500 Hz consisting of at least four oscillations that clearly stand out from the baseline. HFO are biomarkers for epileptogenic brain tissue.

Neuromorphic vs Conventional Processors

Exploitation Technology Transfer and Applications We are now entering the era of neuromorphic intelligence in which dedicated cognitive "chiplets" will be used to provide intelligence to a multitude of edge-computing devices.

The Perfect "recipe" for Fabricating Neuromorphic Intelligence Devices

• "Mix" populations of mixed-signal silicon neurons and synapses.
• "Add" capacitors and volatile memristors for state dynamics and memory traces.
• "Sprinkle" distributed memory elements for parameter storage (SRAM, TCAM, non-volatile memristors).
• "Bake" with asynchronous digital circuits for event-based communication.
• "Serve" with always-on, on-line, self-supervised learning methods.

Conclusion

• Conventional AI increasing power requirements are unsustainable.
• New emerging memory technologies can benefit from massively parallel processing architectures.
• Neuroscience and Machine Learning are uncovering powerful and robust neural processing methods.
• This is the perfect time to follow the neuromorphic engineering approach for starting a hardware revolution.