Notes

DL Challenges - Solved by the Human Brain? Papers: "A Berkeley View of Systems Challenges for AI" & "Complementary roles of basal ganglia and cerebellum in learning and motor control".

• Continual Learning. Being able to learn multiple tasks sequentially.
• Robust Decisions. Taking into account uncertainty and errors in inputs and feedback.
• Explainable Decisions. Understanding network reasoning and learning.
• Security. Shared learning on confidential data.
• Composable AI systems. Combining multiple systems to solve complex tasks. (The human brain is comprised of over 200 different brain areas that all learn differently).
• Unsupervised learning of useful data representations.
• Fast learning and generalization from a few data examples. (There are only 10ˆ7 seconds in a year, so a child would need to ask one question very second of his life to receive a comparable volume of labeled data. In fact, there is a mismatch between the available pool of labeled data and how quickly children learn. Thus, children cannot only rely on supervised algorithms to learn to categorize objects.

The Human Brain as Universal Learning Machine Is the brain a Universal Learning Machine? A species using the mixed strategy may thrive if that strategy achieves a higher asymptotic level of performance.

Papers: "Deep Learning: A Critical Appraisal" & "A Critique of Pure Learning: What Artificial Neural Networks can Learn from Animal Brains" & "A Path Towards Autonomous Machine Intelligence".

The C. Elegans genome stores neuronal wiring! The simple worm C. Elegans, for example, has 302 neurons and about 7000 synapses and in each individual of an inbred strain, the wiring pattern is exactly the same. (Chen et al., 2006).

The Human Brain is mostly Learned. The human brain has about 10ˆ11 neurons, and more than 10ˆ3 synapses per neuron. Specifying a connection target requires about log_2 10ˆ11 + 35 bits/synapse. Thus, it would take about 3.5 x 10ˆ15 bits (approx. 400 TB) to specify all 10ˆ14 connections in the brain.

How did Human Intelligence Emerge? The brain capacity has been constantly increasing during our evolution. Paper: "The Evolution of Intelligence in Mammalian Carnivores".

The Mammalian Neocortex - The Soul of Human Intelligence

• It is the top layer of the cerebral hemispheres, 2-4 mm thick, and made up of six layers, labelled I to VI.
• The neocortex is part of the cerebral cortex (along with the archiocortex and paleocortex, which belong to the limbic system).
• It is involved in higher functions such as sensory perception, generation of motor commands, spatial reasoning, conscious thought, and language.
• The neocortex consists of grey matter surrounding the deeper white matter of the cerebrum.
• While the neocortex is smaller and smoother in rats and some other small mammals, it has deep grooves (sulci) and wrinkles (gyri) in primates and several other mammals. These folds serve to increase the area of the neocortex considerably.
• In humans the neocortex accounts for about 76% of the brain's volume.

Modern AI was inspired by the Neocortex.

• Hierarchical Information Processing in the Neocortex

• The Canonical Cortical Circuit

Paper: "Neuronal Circuits of the Neocortex".

Hubel & Wiesel Experiments in the late 50s

• Orientation and Direction Selective Neurons in Cortex.

The McCulloch and Pitts Neuron (MCP) was inspired by Cortical Neurons. The Perceptron was developed based on the MCP Neuron by Frank Rosenblatt in 1957. MCP-Neuron integrates only binary values. While the Perceptron integrates non-boolean values where every value is associated with a weight.

The Perceptron as Feature Detector.

Solving Complex Classification Tasks with MLPs

Parallels between Artificial and Biological Networks Papers: "Using goal-driven deep learning models to understand sensory cortex" & "Performance-optimized hierarchical models predict neural responses in higher visual cortex" & "Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation".