.. EpyNN documentation master file, created by
sphinx-quickstart on Tue Jul 6 18:46:11 2021.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
.. toctree::
Dropout - Regularization
===============================
Source files in ``EpyNN/epynn/dropout/``.
See `Appendix - Notations`_ for mathematical conventions.
.. _Appendix - Notations: glossary.html#notations
Layer architecture
------------------------------
.. image:: _static/Dropout/Dropout-01.svg
:alt: Dropout
A *Dropout* layer *k* is an object used for *regularization* of a Neural Network model and aiming at diminishing the *overfitting* of the model to the training data. It takes a *drop_prob* and *axis* argument upon instantiation. The first describes the probability to drop - set to zero - input values forwarded to the next layer *k+1* while the latter selects the axes on which the operation should be applied.
.. autoclass:: epynn.dropout.models.Dropout
:show-inheritance:
Shapes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. automethod:: epynn.dropout.models.Dropout.compute_shapes
.. literalinclude:: ./../epynn/dropout/parameters.py
:pyobject: dropout_compute_shapes
:language: python
Within a *Dropout* layer, shapes of interest include:
* Input *X* of shape *(m, ...)* with *m* equal to the number of samples. The number of input dimensions is unknown *a priori*.
* The number of features *n* per sample can still be determined formally: it is equal to the size of the input *X* divided by the number of samples *m*.
* The shape of the dropout mask *D* defined by the user-defined axes *a*.
Note that:
* The *Dropout* operation is applied on all axis by defaults. Given an input of shape *(m, ...)*, the shape of the dropout mask is therefore identical with default settings.
* For input *X* of shape *(m, n)* and *axis=(1)*, the shape of *D* is equal to *(m, 1)*. The *Dropout* operation will set to zero entire columns instead of single values but still with respect to *drop_prob*.
Forward
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. automethod:: epynn.dropout.models.Dropout.forward
.. literalinclude:: ./../epynn/dropout/forward.py
:pyobject: dropout_forward
The forward propagation function in a *Dropout* layer *k* includes:
* (1): Input *X* in current layer *k* is equal to the output *A* of previous layer *k-1*.
* (2): Generate the dropout mask *D* from random uniform distribution.
* (3): Evaluate each value in *D* against the dropout probability *d*. Values in *D* lower than *d* are set to *0* while others are set to *1*.
* (4): *A* is the product between input *X* and dropout mask *D*. In words, values in *A* are set to *0* with respect to dropout probability *d*.
* (5): Values in *A* are scaled-up with respect to fraction of data points that were set to zero by the dropout mask *D*.
.. math::
\begin{alignat*}{2}
& x^{k}_{mn} &&= a^{\km}_{mn} \tag{1} \\
\\
& Let&&D~be~a~matrix~of~the~same~dimensions~as~x^{k}_{mn}: \\
\\
& D &&= [d_{mn}] \in \mathbb{R}^{M \times N}, \\
& &&\forall_{m,n} \in \{1,...M\} \times \{1,...N\} \\
& && d_{mn} \sim \mathcal{U}[0,1] \tag{2} \\
\\
& D' &&= [d_{mn}'] \in \mathbb{R}^{M \times N}, d \in [0,1] \\
& &&\forall_{m,n} \in \{1,...M\} \times \{1,...N\} \\
& && d_{mn}' =
\begin{cases}
1, & d_{mn} > d \\
0, & d_{mn} \le d
\end{cases} \tag{3} \\
\\
& a^{k}_{mn} &&= x^{k}_{mn} * D' \\
& &&~~~* \frac{1}{(1-d)} \tag{4}
\end{alignat*}
Backward
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. automethod:: epynn.dropout.models.Dropout.backward
.. literalinclude:: ./../epynn/dropout/backward.py
:pyobject: dropout_backward
The backward propagation function in a *Dropout* layer *k* includes:
* (1): *dA* the gradient of the loss with respect to the output of forward propagation *A* for current layer *k*. It is equal to the gradient of the loss with respect to input of forward propagation for next layer *k+1*.
* (2): The gradient of the loss *dX* with respect to the input of forward propagation *X* for current layer *k* is equal to the product between *dA* and the dropout mask *D*.
* (3): Values in *dX* are scaled-up with respect to fraction of data point that were set to zero by the dropout mask *D*.
.. math::
\begin{alignat*}{2}
& \delta^{\kp}_{mn} &&= \frac{\partial \mathcal{L}}{\partial a^{k}_{mn}} = \frac{\partial \mathcal{L}}{\partial x^{\kp}_{mn}} \tag{1} \\
& \delta^{k}_{mn} &&= \frac{\partial \mathcal{L}}{\partial x^{k}_{mn}} = \frac{\partial \mathcal{L}}{\partial a^{\km}_{mn}} = \frac{\partial \mathcal{L}}{\partial a^{k}_{mn}} * D'^{k}_{mn} * \frac{1}{(1-d)} \tag{2} \\
\end{alignat*}
Gradients
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. automethod:: epynn.dropout.models.Dropout.compute_gradients
The *Dropout* layer is not a *trainable* layer. It has no *trainable* parameters such as weight *W* or bias *b*. Therefore, there is no parameters gradients to compute.
Live examples
------------------------------
* `Protein Modification - Flatten-(Dense)n with Dropout`_
* `Protein Modification - LSTM(sequence=True)-(Dense)n with Dropout`_
* `Dummy time - Flatten-(Dense)n with Dropout`_
* `Author music - Flatten-(Dense)n-with-Dropout`_
* `Author music - RNN(sequences=True)-Flatten-(Dense)n with Dropout`_
* `Author music - GRU(sequences=True)-Flatten-(Dense)n with Dropout`_
* `Dummy image - Flatten-(Dense)n with Dropout`_
* `MNIST Database - Flatten-(Dense)n with Dropout`_
You may also like to browse all `Network training examples`_ provided with EpyNN.
.. _Network training examples: run_examples.html
.. _Protein Modification - Flatten-(Dense)n with Dropout: epynnlive/ptm_protein/train.html#Flatten-(Dense)n-with-Dropout
.. _Protein Modification - LSTM(sequence\=True)-(Dense)n with Dropout: epynnlive/ptm_protein/train.html#LSTM(sequence=True)-(Dense)n-with-Dropout
.. _Dummy time - Flatten-(Dense)n with Dropout: epynnlive/dummy_time/train.html#Flatten-(Dense)n-with-Dropout
.. _Author music - Flatten-(Dense)n-with-Dropout: epynnlive/author_music/train.html#Flatten-(Dense)n-with-Dropout
.. _Author music - RNN(sequences\=True)-Flatten-(Dense)n with Dropout: epynnlive/author_music/train.html#RNN(sequences=True)-Flatten-(Dense)n-with-Dropout
.. _Author music - GRU(sequences\=True)-Flatten-(Dense)n with Dropout: epynnlive/author_music/train.html#GRU(sequences=True)-Flatten-(Dense)n-with-Dropout
.. _Dummy image - Flatten-(Dense)n with Dropout: epynnlive/dummy_image/train.html#Flatten-(Dense)n-with-Dropout
.. _MNIST Database - Flatten-(Dense)n with Dropout: epynnlive/captcha_mnist/train.html#Flatten-(Dense)n-with-Dropout