Notes

The Deep Network Revolution Although the first successes of ANNs were first demonstrated in the 1980's they only started to outperform classical optimization and engineering approaches from 2009 on. In 2011, CNNs trained using backpropagation on GPUs achieved for the first time superhuman performance in a visual pattern recognition contest.

Deep Networks Galore

• As CNNs and DNNs outperformed classical approaches, many research groups started to extend and optimize them.
• The AI field is now (mostly) dominated by attempts to improve accuracy on standard benchmarks, by scaling up network size and parameter count.
• GPT-3 is a network with 175-billion parameters. It's memory size is exceeding 350GB, and training it requires an estimated $12 million.

Problems and Limitations of AI

The Neuromorphic Engineering Approach Methods

• Study fundamental neuroscience, principles of neuro-physiology, neuro-anatomy, theory of computation, electrical engineering, ...
• Exploit the physics of silicon to reproduce the bio-physics of neural systems, using subthreshold analog VLSI circuits.
• Develop distributed multi-core spiking architectures using asynchronous digital VLSI circuits.
• Build real-time autonomous cognitive agents able to carry out behavioral tasks in complex environment.

Neuromorphic Agents When building processing chips, one can be inspired from life-forms that have brain and use them in every-day tasks, like mammals, insects etc. The physical properties of computational elements in brains can also be used as a reference from the computational resource usage standpoint, for example a bee's brain has a weight of 1 mg, a volume of 1 mm3 and manages to squeeze almost 1 million neurons in it. The energy per operation is approximated to 10−15J/spike. Functional principles used by these life-forms can be used in the development of neuromorphic agents. Some of these are:

• Exploiting physical space in the most efficient way possible.
• Let time represent itself.
• Use both analog and digital computing elements.
• Exploit non-linearities and temporal dynamics.
• Leverage noise, variability and stochasticity.
• Minimize wiring (maximize local connectivity).
• Maximize fine grain parallelism (no time-multiplexing).
• Co-localize memory and computation.
• Use physical time (no clocks).
• Match dynamics and time-constants.
• Optimize for processing complex spatio-temporal (dynamic and noisy) signals.
• Re-use computational principles for sensory processing, motor control and cognitive computing.

Brain-Inspired Computing: A Radical Paradigm Shift Exploit Physical Space

• Use parallel arrays of processing elements.
• Maximize fine grain parallelism (no time-multiplexing).
• To co-localize memory and computation.
• Use passive circuits, data-drive, in continuous time.
• Exploit all the properties of transistors and memristors.

Let Time Represent Itself

• For interacting with the environment in real-time.
• To match the circuit time constants to the input signal dynamics.
• For inherently synchronizing with the real-world natural events.
• To process sensory signals efficiently.