Neural Network - Model
Source files in EpyNN/epynn/network/
.
In most Python implementations of Neural Network frameworks, the top-level object is the Network itself.
The EpyNN Model (epynn.network.models.EpyNN
) is the top-level object which represents the Neural Network designed by the end-user, as a whole.
The design of a Neural Network represents a list of second-level objects called layers, each type of layer having its own architecture.
EpyNN Model
The EpyNN model of neural network network is defined by:
An embedding - or input - layer in first position in the list of layers.
An output layer in last position in the list of layers.
Optional hidden layers.
A loss function which is the error function to be minimized during the training phase.
The training phase of a Neural Network can be briefly summarized such as:
Forward propagation: input data X in the embedding layer are propagated forward and transformed through each layer of the network to output the predicted values of interest A.
Loss evaluation: Predicted values A are compared with target values Y through the loss function (i. e. Mean Squared Error).
Backward propagation: The error is propagated backward through the network and each layer internally computes gradients further used to adjust the trainable parameters.
See below for more details.
- class epynn.network.models.EpyNN(layers, name='EpyNN')[source]
Definition of a Neural Network prototype following the EpyNN scheme.
- Parameters
- __init__(layers, name='EpyNN')[source]
Initialize instance variable attributes.
- Variables
layers (list[Object]) – Network architecture.
embedding (
epynn.embedding.models.Embedding
) – Embedding layer.ts (int) – Timestamp identifier.
uname (str) – Network unique identifier.
initialized (bool) – Model initialization state.
Forward
- EpyNN.forward(X)[source]
Wrapper for
epynn.network.forward.model_forward()
.
- Parameters
X (
numpy.ndarray
) – Set of sample features.- Returns
Output of forward propagation through all layers in the Network.
- Return type
numpy.ndarray
def model_forward(model, X): """Forward propagate input data from input to output layer. """ # By convention A = X # Iterate over layers for layer in model.layers: # For learning rate schedule layer.e = model.e # Layer returns A - layer.fs, layer.fc A = layer.forward(A) return A # To derivative of loss functionThe forward propagation for a Neural Network model is straightforward. Given the
layers
attribute representing the list of layers contained in the model, afor
loop statement is used to iterate overmodel.layers
. For each layer, theforward()
method is called. Therefore, the forward propagation for one model is defined by successive forward propagation for each layer within the model.
Backward
- EpyNN.backward(dA)[source]
Wrapper for
epynn.network.backward.model_backward()
.
- Parameters
dA (
numpy.ndarray
) – Derivative of the loss function with respect to the output of forward propagation.def model_backward(model, dA): """Backward propagate error gradients from output to input layer. """ # By convention dX = dA # Iterate over reversed layers for layer in reversed(model.layers): # Layer returns dL/dX (dX) - layer.bs, layer.bc dX = layer.backward(dX) # Update values in layer.g layer.compute_gradients() # Update values in layer.p layer.update_parameters() return NoneFor the backward propagation we use a
for
loop statement to iterate backward overmodel.layers
. For each layer, thebackward()
method is called, followed by thecompute_gradients()
andupdate_parameters()
methods.
Initialize
- EpyNN.initialize(loss='MSE', se_hPars={'ELU_alpha': 1, 'LRELU_alpha': 0.3, 'cycle_descent': 0, 'cycle_epochs': 0, 'decay_k': 0, 'learning_rate': 0.1, 'schedule': 'steady', 'softmax_temperature': 1}, metrics=['accuracy'], seed=None, params=True, end='\n')[source]
Wrapper for
epynn.network.initialize.model_initialize()
. Perform a dry epoch including all but not the parameters update step.
- Parameters
loss (str, optional) – Loss function to use for training, defaults to ‘MSE’. See
epynn.commons.loss
for built-in functions.se_hPars (dict[str: float or str], optional) – Global hyperparameters, defaults to
epynn.settings.se_hPars
. If local hyperparameters were assigned to one layer, these remain unchanged.metrics (list[str], optional) – Metrics to monitor and print on terminal report or plot, defaults to [‘accuracy’]. See
epynn.commons.metrics
for built-in metrics. Note that it also accept loss functions string identifiers.seed (int or NoneType, optional) – Reproducibility in pseudo-random procedures.
params (bool, optional) – Layer parameters initialization, defaults to True.
end (str in ['n', 'r'], optional) – Whether to print every line for initialization steps or overwrite, default to n.
See Model Hyperparameters for details about
se_hPars
argument.Note that:
Each layer is assigned with independent pseudo-random number generator such as
layer.np_rng = np.random.default_rng(seed=seed_layer)
. When set to anint
value, the seed parameter is incremented by one after each generator initialization.There is a single loss function that can be assigned to one model to actually drive the regression.
The
metrics
argument can be provided with a list of metrics but also with loss functions to compute the associated cost in a purely indicative manner. The regression is never driven by the value of themetrics
argument whether or not it is provided with a loss function.
Training
- EpyNN.train(epochs, verbose=None, init_logs=True)[source]
Wrapper for
epynn.network.training.model_training()
. Apart, it computes learning rate along learning epochs.
- Parameters
def model_training(model): """Perform the training of the Neural Network. :param model: An instance of EpyNN network. :type model: :class:`epynn.network.models.EpyNN` """ # Iterate over training epochs for model.e in range(model.e, model.epochs): # Shuffle dtrain and prepare new batches model.embedding.training_batches() # Iterate over training batches for batch in model.embedding.batch_dtrain: # Pass through every layer.forward() methods A = model.forward(batch.X) # Compute derivative of loss dA = model.training_loss(batch.Y, A, deriv=True) # Pass through every layer.backward() methods model.backward(dA) # Accuracy and cost for batch model.batch_report(batch, A) # Selected metrics and costs for dsets model.evaluate() # Tabular report for dsets model.report() return NoneThe full training procedure for one model includes:
A
for
loop statement iterating over a user-defined number of training epochs.For each epoch, training batches are built from the training set.
For each epoch, a
for
loop statement iterates over training batches.For each batch, the
model.forward()
is computed. The training loss derivative is then computed by comparison of the true values Y with the predicted values A. Themodel.backward()
method is then called. Accuracy and training loss are then reported for the current batch.For each epoch and once all training batches have been processed, model evaluation is achieved using the
model.evaluate()
method and results are reported report using themodel.report()
method.
- EpyNN.evaluate()[source]
Wrapper for
epynn.network.evaluate.model_evaluate()
. Good spot for further implementation of early stopping procedures.Compute metrics including cost for dsets. Metrics of interest are provided as argument when calling the
model.initialize()
method.
- EpyNN.report()[source]
Wrapper for
epynn.network.report.model_report()
.Report selected metrics for dsets at current epoch.
- EpyNN.plot(pyplot=True, path=None)[source]
Wrapper for
epynn.commons.plot.pyplot_metrics()
. Plot metrics from model training.
- Parameters
pyplot (bool, optional) – Plot of results on GUI using matplotlib.
path (str or bool or NoneType, optional) – Write matplotlib plot, defaults to None which writes in the plots subdirectory created from
epynn.commons.library.configure_directory()
. To not write the plot at all, set to False.
- EpyNN.write(path=None)[source]
Write model on disk.
- Parameters
path (str or NoneType, optional) – Path to write the model on disk, defaults to None which writes in the models subdirectory created from
epynn.commons.library.configure_directory()
.If the model you trained fits your requirements, write on disk to use it later.
Prediction
- EpyNN.predict(X_data, X_encode=False, X_scale=False)[source]
Perform prediction of label from unlabeled samples in dataset.
- Parameters
- Returns
Data embedding and output of forward propagation.
- Return type
def predict(self, X_data, X_encode=False, X_scale=False): """Perform prediction of label from unlabeled samples in dataset. """ X_data = np.array(X_data) if X_encode: # One-hot encoding using embedding layer cache element_to_idx = self.embedding.e2i elements_size = self.embedding.d['e'] X_data = encode_dataset(X_data, element_to_idx, elements_size) if X_scale: # Array-wide normalization in [0, 1] X_data = scale_features(X_data) dset = dataSet(X_data) # Predict dset.A = self.forward(dset.X) # Check label encoding encoded = (self.embedding.dtrain.Y.shape[1] > 1) # Make decisions dset.P = np.argmax(dset.A, axis=1) if encoded else np.around(dset.A) return dsetA model previously written on disk can be read from it using the
epynn.commons.library.read_model()
function.Prediction using a pre-trained model includes:
Data pre-processing: if data used for training were one-hot encoded, they must also be encoded for prediction. Similarly, if a global scaling within [0, 1] was applied during the training phase, it should be applied to the data before prediction. Note that scaling should be used with caution because data range may be unrelated between training data and samples for prediction.
Data embedding into the dataSet object: This object is convenient because it has multiple attributes to store data provided by user as well as relevant outputs from prediction protocol.
Prediction: This is equal to a single
model.forward()
pass. Probabilities are stored indset.A
while decisions are stored indset.P
.
Model Hyperparameters
Defaults hyperparameters settings upon model initialization. When loaded from epynn.network.models.EpyNN.initialize()
these are global hyperparameters which apply on every layer with no anterior assignment. See Layer Hyperparameters for details on local hyperparameters assignment and more in-depth description.
Also note that active hyperparameters can be accessed from model.layers[i].se_hPars
.
- epynn.settings.se_hPars
Hyperparameters dictionary settings.
Set hyperparameters for model and layer.
se_hPars = {
# Schedule learning rate
'learning_rate': 0.1,
'schedule': 'steady',
'decay_k': 0,
'cycle_epochs': 0,
'cycle_descent': 0,
# Tune activation function
'ELU_alpha': 1,
'LRELU_alpha': 0.3,
'softmax_temperature': 1,
}
"""Hyperparameters dictionary settings.
Set hyperparameters for model and layer.
"""
Because EpyNN is meant to be friendly for less experienced peoples in programming and/or Neural Networks, we made the choice to feed these hyperparameters as dict
rather than unfolding it in a long sequence of arguments.
Still, this setup allows the following for the advanced user:
Save your best
se_hPars
at the location of your choice.Implement specific hyperparameters such as
se_hPars_Dense
in a separate file and import to provide these_hPars
argument upon layer instantiation.Implement program-wide new hyperparameters by adding the corresponding
key: value
pair inepynn.settings.se_hPars
and use it for hyperparameters assignment.