Introduction
The aim of this section is to present basics of Neural Networks in a way that will be accessible for most visitors.
Basic Concepts and Limitations
What is a Neural Network?
In the literature, it is often explained that Artificial Neural Networks (ANN) evolved from the idea of simulating the human brain [1].
This idea may originate from McCulloch & Pitts who published in 1943 a paper entitled “A logical calculus of the ideas immanent in nervous activity” [2] where they developed the theory according to which “Because of the ‘all-or-none’ character of nervous activity, neural events and the relations among them can be treated by means of propositional logic.”
In common language, this means that upon integration and weighting of various stimuli by a given biological neuron, this same neuron may spike or may not spike, thus propagating forward the information to the next neuron.
In ANNs, we would translate by saying that a node within the network will integrate inputs, weight them and pass the product through a step_function(input * weight) which propagates forward True or False to the next node.
Why use Neural Networks?
Neural Networks are claimed to be universal function approximators [3].
In the scope of Supervised Machine Learning (SML), it means that ANNs can theoretically determine any function which will approximate the relationship between inputs (X) and output (Y) known a priori and describing a given sample.
Importantly, ANNs deal with n-dimensional input features. It means that one sample is described by one or more features to which a label is associated.
Included in a training data set, a relatively large number of samples is provided to the Neural Network which will iterate over a user-provided number of epochs or iterations.
Provided with a learning rule, which is generally composed of a learning rate and a cost function to be minimized, ANNs can auto-adjust their trainable parameters to find one function which will make the link between features and labels.
The ability of ANNs to detect patterns in sample features that are relevant to associated labels makes them widely used for classification tasks, anomaly detection, time series predictions, and deconvolution among many others.
Daily life examples may include music source separation which is the task of separating a waveform in individual sources corresponding to voices and accompaniments. Such a task was historically challenging but enormous progress was made through deep learning approaches [4]. Recently, a simple python library was published based on ANNs which provides for anyone state-of-the-art results in such tasks [5].
See also
.wav files: Music Authorship.Another example consists of artist-based painting classification. From a set of art- paintings (features) and associated artists (label), researchers could develop an ANN which was effective and achieved state-of-the-art performance in pairing paintings and the corresponding artist [6].
See also
In the daily life of the biochemist, the structural biologist or more generally anyone interested in protein or peptide sequences, the first applications of Neural Networks date from decades ago. As early as 1989, Howard L. Holley and Nobel Prize laureate Mark Karplus published a paper entitled “Protein secondary structure prediction with a neural network” [7]. Since then, the NMR spectroscopist may acknowledge “TALOS+: a hybrid method for predicting protein backbone torsion angles from NMR chemical shifts” [8] which partially relies on Neural Networks to predict angle constrains extensively used in protein structure calculation from NMR data. Finally, the field of protein research is looking forward to recent developments from “Highly accurate protein structure prediction with AlphaFold” [9].
See also
Which limits for Neural Networks?
Artificial Neural Networks (ANNs) are extremely efficient computational models which can solve problems in many fields.
However, there are classical limitations to such models.
Training corpus
In Supervised Machine Learning (SML) based on ANNs, the principle is to provide a training corpus which should contain a relatively large number of samples being representative of the phenomenon of interest.
Overfitting
Overfitting is an error in the regression procedure which happens when the model corresponds too closely to a particular set of data. Said differently, ANNs tend to include outliers in the output model.
Explainability
Because ANNs are nested and non-linear structures with thousands – and up to billions – of parameters, they are difficult to interpret and are often considered as “black boxes” used for performance [10].
Implementation and Vocabulary
Artificial Neural Networks (ANNs) can be implemented in Python.
Python is an object-oriented programming language. This means that abstract objects - equivalent to object prototypes - can be defined in a custom manner by the programmer.
Such prototypes are defined in class, such as:
class MyModel:
def __init__(self):
self.name = 'MyModel'
def my_method(self, msg):
print(msg)
The object related to the class MyModel can be instantiated by calling the corresponding class constructor such as:
my_object = MyModel()
The object my_object is now an instance of the class MyModel.
When the instance my_object is being created, the __init__() method is executed. Later on, other methods defined within the class can be executed from the instantiated object, such as:
my_object.my_method('My Message')
Which will print My Message on the terminal session.
See also
This scheme is the basis for the design of most AI-related libraries, including EpyNN.
Neural Network Model
If willing to define a model object - or object prototype - for a Neural Network, then a minimalist version could be summarized such as:
class MyNetwork:
"""
This is a non-functional summary.
"""
def __init__(self, layers=[]):
self.layers = layers
def training(self, X_data, Y_data):
# Procedure to train the network from labeled data
pass
def predict(self, X_data):
# Procedure to predict label from unlabeled data
pass
The corresponding object instance can be instantiated such as:
my_network = MyNetwork(layers=[input_layer, output_layer])
While the network could be trained such as:
my_network.train(X_train, Y_train)
And further used to predict from unlabeled data, such as:
my_network.predict(X_test)
In short, the Neural Network model contains the training procedure. The actual architecture of the Neural Network is defined within the layers argument passed when calling the class constructor.
The detailed implementation of the EpyNN model for such Neural Network objects is presented in the Neural Network - Model section.
Layer Architecture Model
We have seen that a model of a Neural Network can be instantiated by calling its class constructor provided with a list of layers.
Layers prototypes can also be defined from class, with a minimal example such as:
class MyLayer:
"""
This is a non-functional summary.
"""
def __init__(self):
def forward(self, A):
# Procedure to propagate the signal forward
return A # To next layer
def backward(self, dA):
# Procedure to propagate the error backward
return dA # To previous layer
def update_parameters(self, gradients):
# Procedure to update parameters from gradients
pass
Please follow Base Layer and Template Layer for details about implementation of layers in EpyNN.
Data Model
We have seen how a Neural Network object could be instantiated by calling the corresponding class constructor which takes a list of layers as argument, itself being the custom architecture designed by the end-user.
It is a good practice to define a data model to which X_data and Y_data will be linked, together with dependent outputs generated by the Neural Network during training and prediction.
Such a minimal model may look like:
class MyData:
"""
This is a non-functional summary.
"""
def __init__(self, X_data, Y_data=None):
self.X = X_data
self.Y = Y_data
In practice, such objects may be linked with the input or embedding layer.
The detailed implementation of the dataSet data model in EpyNN is presented in the Data - Model section while the implementation of the Embedding layer model is presented in Embedding layer (Input).
- 1
Jinming Zou, Yi Han, and Sung-Sau So. Overview of artificial neural networks. Artificial Neural Networks, pages 14–22, 2008.
- 2
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115–133, 1943.
- 3
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
- 4
Stefan Uhlich, Marcello Porcu, Franck Giron, Michael Enenkl, Thomas Kemp, Naoya Takahashi, and Yuki Mitsufuji. Improving music source separation based on deep neural networks through data augmentation and network blending. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 261–265. IEEE, 2017.
- 5
Romain Hennequin, Anis Khlif, Felix Voituret, and Manuel Moussallam. Spleeter: a fast and efficient music source separation tool with pre-trained models. Journal of Open Source Software, 5(50):2154, 2020.
- 6
Kai-Lung Hua, Trang-Thi Ho, Kevin-Alfianto Jangtjik, Yu-Jen Chen, and Mei-Chen Yeh. Artist-based painting classification using markov random fields with convolution neural network. Multimedia Tools and Applications, pages 1–24, 2020.
- 7
L Howard Holley and Martin Karplus. Protein secondary structure prediction with a neural network. Proceedings of the National Academy of Sciences, 86(1):152–156, 1989.
- 8
Yang Shen, Frank Delaglio, Gabriel Cornilescu, and Ad Bax. Talos+: a hybrid method for predicting protein backbone torsion angles from nmr chemical shifts. Journal of biomolecular NMR, 44(4):213–223, 2009.
- 9
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, and others. Highly accurate protein structure prediction with alphafold. Nature, pages 1–11, 2021.
- 10
Wojciech Samek and Klaus-Robert Müller. Towards explainable artificial intelligence. In Explainable AI: interpreting, explaining and visualizing deep learning, pages 5–22. Springer, 2019.