Neural Network - Model

Source files in EpyNN/epynn/network/.

In most Python implementations of Neural Network frameworks, the top-level object is the Network itself.

The EpyNN Model (epynn.network.models.EpyNN) is the top-level object which represents the Neural Network designed by the end-user, as a whole.

The design of a Neural Network represents a list of second-level objects called layers, each type of layer having its own architecture.

EpyNN Model

The EpyNN model of neural network network is defined by:

An embedding - or input - layer in first position in the list of layers.
An output layer in last position in the list of layers.
Optional hidden layers.
A loss function which is the error function to be minimized during the training phase.

The training phase of a Neural Network can be briefly summarized such as:

Forward propagation: input data X in the embedding layer are propagated forward and transformed through each layer of the network to output the predicted values of interest A.
Loss evaluation: Predicted values A are compared with target values Y through the loss function (i. e. Mean Squared Error).
Backward propagation: The error is propagated backward through the network and each layer internally computes gradients further used to adjust the trainable parameters.

See below for more details.

class epynn.network.models.EpyNN(layers, name='EpyNN')[source]

Definition of a Neural Network prototype following the EpyNN scheme.

Parameters

layers (list[Object]) – Network architecture.
name (str, optional) – Name of network, defaults to ‘EpyNN’.

__init__(layers, name='EpyNN')[source]

Initialize instance variable attributes.

Variables

layers (list[Object]) – Network architecture.
embedding (epynn.embedding.models.Embedding) – Embedding layer.
ts (int) – Timestamp identifier.
uname (str) – Network unique identifier.
initialized (bool) – Model initialization state.

Forward

EpyNN.forward(X)[source]
Wrapper for epynn.network.forward.model_forward().

Parameters

X (numpy.ndarray) – Set of sample features.

Returns

Output of forward propagation through all layers in the Network.

Return type

numpy.ndarray
def model_forward(model, X):
    """Forward propagate input data from input to output layer.
    """
    # By convention
    A = X

    # Iterate over layers
    for layer in model.layers:

        # For learning rate schedule
        layer.e = model.e

        # Layer returns A - layer.fs, layer.fc
        A = layer.forward(A)

    return A    # To derivative of loss function
The forward propagation for a Neural Network model is straightforward. Given the layers attribute representing the list of layers contained in the model, a for loop statement is used to iterate over model.layers. For each layer, the forward() method is called. Therefore, the forward propagation for one model is defined by successive forward propagation for each layer within the model.

Backward

EpyNN.backward(dA)[source]
Wrapper for epynn.network.backward.model_backward().

Parameters

dA (numpy.ndarray) – Derivative of the loss function with respect to the output of forward propagation.
def model_backward(model, dA):
    """Backward propagate error gradients from output to input layer.
    """
    # By convention
    dX = dA

    # Iterate over reversed layers
    for layer in reversed(model.layers):

        # Layer returns dL/dX (dX) - layer.bs, layer.bc
        dX = layer.backward(dX)

        # Update values in layer.g
        layer.compute_gradients()

        # Update values in layer.p
        layer.update_parameters()

    return None
For the backward propagation we use a for loop statement to iterate backward over model.layers. For each layer, the backward() method is called, followed by the compute_gradients() and update_parameters() methods.

Initialize

EpyNN.initialize(loss='MSE', se_hPars={'ELU_alpha': 1, 'LRELU_alpha': 0.3, 'cycle_descent': 0, 'cycle_epochs': 0, 'decay_k': 0, 'learning_rate': 0.1, 'schedule': 'steady', 'softmax_temperature': 1}, metrics=['accuracy'], seed=None, params=True, end='\n')[source]

Wrapper for epynn.network.initialize.model_initialize(). Perform a dry epoch including all but not the parameters update step.

Parameters

loss (str, optional) – Loss function to use for training, defaults to ‘MSE’. See epynn.commons.loss for built-in functions.

se_hPars (dict[str: float or str], optional) – Global hyperparameters, defaults to epynn.settings.se_hPars. If local hyperparameters were assigned to one layer, these remain unchanged.

metrics (list[str], optional) – Metrics to monitor and print on terminal report or plot, defaults to [‘accuracy’]. See epynn.commons.metrics for built-in metrics. Note that it also accept loss functions string identifiers.

seed (int or NoneType, optional) – Reproducibility in pseudo-random procedures.

params (bool, optional) – Layer parameters initialization, defaults to True.

end (str in ['n', 'r'], optional) – Whether to print every line for initialization steps or overwrite, default to n.

See Model Hyperparameters for details about se_hPars argument.

Note that:

Each layer is assigned with independent pseudo-random number generator such as layer.np_rng = np.random.default_rng(seed=seed_layer). When set to an int value, the seed parameter is incremented by one after each generator initialization.

There is a single loss function that can be assigned to one model to actually drive the regression.

The metrics argument can be provided with a list of metrics but also with loss functions to compute the associated cost in a purely indicative manner. The regression is never driven by the value of the metrics argument whether or not it is provided with a loss function.

Training

EpyNN.train(epochs, verbose=None, init_logs=True)[source]
Wrapper for epynn.network.training.model_training(). Apart, it computes learning rate along learning epochs.

Parameters

epochs (int) – Number of training iterations.

verbose (int or NoneType, optional) – Print logs every Nth epochs, defaults to None which sets to every tenth of epochs.

init_logs (bool, optional) – Print data, architecture and hyperparameters logs, defaults to True.
def model_training(model):
    """Perform the training of the Neural Network.

    :param model: An instance of EpyNN network.
    :type model: :class:`epynn.network.models.EpyNN`
    """
    # Iterate over training epochs
    for model.e in range(model.e, model.epochs):

        # Shuffle dtrain and prepare new batches
        model.embedding.training_batches()

        # Iterate over training batches
        for batch in model.embedding.batch_dtrain:

            # Pass through every layer.forward() methods
            A = model.forward(batch.X)

            # Compute derivative of loss
            dA = model.training_loss(batch.Y, A, deriv=True)

            # Pass through every layer.backward() methods
            model.backward(dA)

            # Accuracy and cost for batch
            model.batch_report(batch, A)

        # Selected metrics and costs for dsets
        model.evaluate()

        # Tabular report for dsets
        model.report()

    return None
The full training procedure for one model includes:

A for loop statement iterating over a user-defined number of training epochs.

For each epoch, training batches are built from the training set.

For each epoch, a for loop statement iterates over training batches.

For each batch, the model.forward() is computed. The training loss derivative is then computed by comparison of the true values Y with the predicted values A. The model.backward() method is then called. Accuracy and training loss are then reported for the current batch.

For each epoch and once all training batches have been processed, model evaluation is achieved using the model.evaluate() method and results are reported report using the model.report() method.
EpyNN.evaluate()[source]

Wrapper for epynn.network.evaluate.model_evaluate(). Good spot for further implementation of early stopping procedures.

Compute metrics including cost for dsets. Metrics of interest are provided as argument when calling the model.initialize() method.

EpyNN.report()[source]

Wrapper for epynn.network.report.model_report().

Report selected metrics for dsets at current epoch.

EpyNN.plot(pyplot=True, path=None)[source]

Wrapper for epynn.commons.plot.pyplot_metrics(). Plot metrics from model training.

Parameters

pyplot (bool, optional) – Plot of results on GUI using matplotlib.

path (str or bool or NoneType, optional) – Write matplotlib plot, defaults to None which writes in the plots subdirectory created from epynn.commons.library.configure_directory(). To not write the plot at all, set to False.

EpyNN.write(path=None)[source]

Write model on disk.

Parameters

path (str or NoneType, optional) – Path to write the model on disk, defaults to None which writes in the models subdirectory created from epynn.commons.library.configure_directory().

If the model you trained fits your requirements, write on disk to use it later.

Prediction

EpyNN.predict(X_data, X_encode=False, X_scale=False)[source]
Perform prediction of label from unlabeled samples in dataset.

Parameters

X_data (list[list[int or float or str]] or numpy.ndarray) – Set of sample features.

X_encode (bool, optional) – One-hot encode sample features, defaults to False.

X_scale (bool, optional) – Normalize sample features within [0, 1] along all axis, default to False.

Returns

Data embedding and output of forward propagation.

Return type

epynn.commons.models.dataSet
    def predict(self, X_data, X_encode=False, X_scale=False):
        """Perform prediction of label from unlabeled samples in dataset.
        """
        X_data = np.array(X_data)

        if X_encode:
            # One-hot encoding using embedding layer cache
            element_to_idx = self.embedding.e2i
            elements_size = self.embedding.d['e']
            X_data = encode_dataset(X_data, element_to_idx, elements_size)

        if X_scale:
            # Array-wide normalization in [0, 1]
            X_data = scale_features(X_data)

        dset = dataSet(X_data)

        # Predict
        dset.A = self.forward(dset.X)

        # Check label encoding
        encoded = (self.embedding.dtrain.Y.shape[1] > 1)

        # Make decisions
        dset.P = np.argmax(dset.A, axis=1) if encoded else np.around(dset.A)

        return dset
A model previously written on disk can be read from it using the epynn.commons.library.read_model() function.

Prediction using a pre-trained model includes:

Data pre-processing: if data used for training were one-hot encoded, they must also be encoded for prediction. Similarly, if a global scaling within [0, 1] was applied during the training phase, it should be applied to the data before prediction. Note that scaling should be used with caution because data range may be unrelated between training data and samples for prediction.

Data embedding into the dataSet object: This object is convenient because it has multiple attributes to store data provided by user as well as relevant outputs from prediction protocol.

Prediction: This is equal to a single model.forward() pass. Probabilities are stored in dset.A while decisions are stored in dset.P.

Model Hyperparameters

Defaults hyperparameters settings upon model initialization. When loaded from epynn.network.models.EpyNN.initialize() these are global hyperparameters which apply on every layer with no anterior assignment. See Layer Hyperparameters for details on local hyperparameters assignment and more in-depth description.

Also note that active hyperparameters can be accessed from model.layers[i].se_hPars.

epynn.settings.se_hPars

Hyperparameters dictionary settings.

Set hyperparameters for model and layer.

se_hPars = {
    # Schedule learning rate
    'learning_rate': 0.1,
    'schedule': 'steady',
    'decay_k': 0,
    'cycle_epochs': 0,
    'cycle_descent': 0,
    # Tune activation function
    'ELU_alpha': 1,
    'LRELU_alpha': 0.3,
    'softmax_temperature': 1,
}
"""Hyperparameters dictionary settings.

Set hyperparameters for model and layer.
"""

Because EpyNN is meant to be friendly for less experienced peoples in programming and/or Neural Networks, we made the choice to feed these hyperparameters as dict rather than unfolding it in a long sequence of arguments.

Still, this setup allows the following for the advanced user:

Save your best se_hPars at the location of your choice.
Implement specific hyperparameters such as se_hPars_Dense in a separate file and import to provide the se_hPars argument upon layer instantiation.
Implement program-wide new hyperparameters by adding the corresponding key: value pair in epynn.settings.se_hPars and use it for hyperparameters assignment.