The aim of this section is to present basics of Neural Networks in a way that will be accessible for most visitors.
Basic Concepts and Limitations¶
What is a Neural Network?¶
In the literature, it is often explained that Artificial Neural Networks (ANN) evolved from the idea of simulating the human brain .
This idea may originate from McCulloch & Pitts who published in 1943 a paper entitled “A logical calculus of the ideas immanent in nervous activity”  where they developed the theory according to which “Because of the ‘all-or-none’ character of nervous activity, neural events and the relations among them can be treated by means of propositional logic.”
In common language, this means that upon integration and weighting of various stimuli by a given biological neuron, this same neuron may spike or may not spike, thus propagating forward the information to the next neuron.
In ANNs, we would translate by saying that a node within the network will integrate inputs, weight them and pass the product through a
step_function(input * weight) which propagates forward
False to the next node.
Why use Neural Networks?¶
Neural Networks are claimed to be universal function approximators .
In the scope of Supervised Machine Learning (SML), it means that ANNs can theoretically determine any function which will approximate the relationship between inputs (X) and output (Y) known a priori and describing a given sample.
Importantly, ANNs deal with n-dimensional input features. It means that one sample is described by one or more features to which a label is associated.
Included in a training data set, a relatively large number of samples is provided to the Neural Network which will iterate over a user-provided number of epochs or iterations.
Provided with a learning rule, which is generally composed of a learning rate and a cost function to be minimized, ANNs can auto-adjust their trainable parameters to find one function which will make the link between features and labels.
The ability of ANNs to detect patterns in sample features that are relevant to associated labels makes them widely used for classification tasks, anomaly detection, time series predictions, and deconvolution among many others.
Daily life examples may include music source separation which is the task of separating a waveform in individual sources corresponding to voices and accompaniments. Such a task was historically challenging but enormous progress was made through deep learning approaches . Recently, a simple python library was published based on ANNs which provides for anyone state-of-the-art results in such tasks .
Another example consists of artist-based painting classification. From a set of art- paintings (features) and associated artists (label), researchers could develop an ANN which was effective and achieved state-of-the-art performance in pairing paintings and the corresponding artist .
In the daily life of the biochemist, the structural biologist or more generally anyone interested in protein or peptide sequences, the first applications of Neural Networks date from decades ago. As early as 1989, Howard L. Holley and Nobel Prize laureate Mark Karplus published a paper entitled “Protein secondary structure prediction with a neural network” . Since then, the NMR spectroscopist may acknowledge “TALOS+: a hybrid method for predicting protein backbone torsion angles from NMR chemical shifts”  which partially relies on Neural Networks to predict angle constrains extensively used in protein structure calculation from NMR data. Finally, the field of protein research is looking forward to recent developments from “Highly accurate protein structure prediction with AlphaFold” .
Which limits for Neural Networks?¶
Artificial Neural Networks (ANNs) are extremely efficient computational models which can solve problems in many fields.
However, there are classical limitations to such models.
In Supervised Machine Learning (SML) based on ANNs, the principle is to provide a training corpus which should contain a relatively large number of samples being representative of the phenomenon of interest.
Overfitting is an error in the regression procedure which happens when the model corresponds too closely to a particular set of data. Said differently, ANNs tend to include outliers in the output model.
Because ANNs are nested and non-linear structures with thousands – and up to billions – of parameters, they are difficult to interpret and are often considered as “black boxes” used for performance .
Implementation and Vocabulary¶
Artificial Neural Networks (ANNs) can be implemented in Python.
Python is an object-oriented programming language. This means that abstract objects - equivalent to object prototypes - can be defined in a custom manner by the programmer.
Such prototypes are defined in
class, such as:
class MyModel: def __init__(self): self.name = 'MyModel' def my_method(self, msg): print(msg)
The object related to the class
MyModel can be instantiated by calling the corresponding class constructor such as:
my_object = MyModel()
my_object is now an instance of the class
When the instance
my_object is being created, the
__init__() method is executed. Later on, other methods defined within the class can be executed from the instantiated object, such as:
Which will print
My Message on the terminal session.
This scheme is the basis for the design of most AI-related libraries, including EpyNN.
Neural Network Model¶
If willing to define a model object - or object prototype - for a Neural Network, then a minimalist version could be summarized such as:
class MyNetwork: """ This is a non-functional summary. """ def __init__(self, layers=): self.layers = layers def training(self, X_data, Y_data): # Procedure to train the network from labeled data pass def predict(self, X_data): # Procedure to predict label from unlabeled data pass
The corresponding object instance can be instantiated such as:
my_network = MyNetwork(layers=[input_layer, output_layer])
While the network could be trained such as:
And further used to predict from unlabeled data, such as:
In short, the Neural Network model contains the training procedure. The actual architecture of the Neural Network is defined within the
layers argument passed when calling the class constructor.
The detailed implementation of the
EpyNN model for such Neural Network objects is presented in the Neural Network - Model section.
Layer Architecture Model¶
We have seen that a model of a Neural Network can be instantiated by calling its class constructor provided with a list of layers.
Layers prototypes can also be defined from
class, with a minimal example such as:
class MyLayer: """ This is a non-functional summary. """ def __init__(self): def forward(self, A): # Procedure to propagate the signal forward return A # To next layer def backward(self, dA): # Procedure to propagate the error backward return dA # To previous layer def update_parameters(self, gradients): # Procedure to update parameters from gradients pass
We have seen how a Neural Network object could be instantiated by calling the corresponding class constructor which takes a list of layers as argument, itself being the custom architecture designed by the end-user.
It is a good practice to define a data model to which
Y_data will be linked, together with dependent outputs generated by the Neural Network during training and prediction.
Such a minimal model may look like:
class MyData: """ This is a non-functional summary. """ def __init__(self, X_data, Y_data=None): self.X = X_data self.Y = Y_data
In practice, such objects may be linked with the input or embedding layer.
Jinming Zou, Yi Han, and Sung-Sau So. Overview of artificial neural networks. Artificial Neural Networks, pages 14–22, 2008.
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115–133, 1943.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
Stefan Uhlich, Marcello Porcu, Franck Giron, Michael Enenkl, Thomas Kemp, Naoya Takahashi, and Yuki Mitsufuji. Improving music source separation based on deep neural networks through data augmentation and network blending. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 261–265. IEEE, 2017.
Romain Hennequin, Anis Khlif, Felix Voituret, and Manuel Moussallam. Spleeter: a fast and efficient music source separation tool with pre-trained models. Journal of Open Source Software, 5(50):2154, 2020.
Kai-Lung Hua, Trang-Thi Ho, Kevin-Alfianto Jangtjik, Yu-Jen Chen, and Mei-Chen Yeh. Artist-based painting classification using markov random fields with convolution neural network. Multimedia Tools and Applications, pages 1–24, 2020.
L Howard Holley and Martin Karplus. Protein secondary structure prediction with a neural network. Proceedings of the National Academy of Sciences, 86(1):152–156, 1989.
Yang Shen, Frank Delaglio, Gabriel Cornilescu, and Ad Bax. Talos+: a hybrid method for predicting protein backbone torsion angles from nmr chemical shifts. Journal of biomolecular NMR, 44(4):213–223, 2009.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, and others. Highly accurate protein structure prediction with alphafold. Nature, pages 1–11, 2021.
Wojciech Samek and Klaus-Robert Müller. Towards explainable artificial intelligence. In Explainable AI: interpreting, explaining and visualizing deep learning, pages 5–22. Springer, 2019.