Content for  TR 22.874  Word version:  18.2.0

Within the ML field, there is an area that is often referred to as brain-inspired computation, which is a program aiming to emulate some aspects of how we understand the brain to operate. Since it is believed that the main computational elements a human brain are 86 billion neurons, the two subareas of brain-inspired computation are both inspired by the architecture of a neuron [25], as shown in Figure A.2-1 (a). Compared to spiking computing approaches, e.g. [3], the more popular ML approaches are using "neural network" as the model. Neural networks (NN) take their inspiration from the notion that a neuron's computation involves a weighted sum of the input values. But instead of simply outputting the weighted sum, a NN applies a nonlinear function to generate an output only if the inputs cross some threshold, as shown in Figure A.2-1 (a). Figure A.2-1 (b) shows a diagrammatic picture of a computational neural network. The neurons in the input layer receive some values and propagate them to the neurons in the middle layer of the network, which is also called a "hidden layer". The weighted sums from one or more hidden layers are ultimately propagated to the output layer, which presents the final outputs of the network [25].

Neural networks having more than three layers, i.e., more than one hidden layer are called deep neural networks (DNN). In contrast to the conventional shallow-structured NN architectures, DNNs, also referred to as deep learning, made amazing breakthroughs since 2010s in many essential application areas because they can achieve human-level accuracy or even exceed human accuracy. Deep learning techniques use supervised and/or unsupervised strategies to automatically learn hierarchical representations in deep architectures for classification [26]. With a large number of hidden layers, the superior performance of DNNs comes from its ability to extract high-level features from raw sensory data after using statistical learning over a large amount of data to obtain an effective representation of an input space [25]. In recent years, thanks to the big data obtained from the real world, the rapidly increased computation capacity and continuously-evolved algorithms, DNNs have become the most popular ML models for many AI applications.