.. EpyNN documentation master file, created by sphinx-quickstart on Tue Jul 6 18:46:11 2021. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. .. toctree:: Dropout - Regularization =============================== Source files in ``EpyNN/epynn/dropout/``. See `Appendix - Notations`_ for mathematical conventions. .. _Appendix - Notations: glossary.html#notations Layer architecture ------------------------------ .. image:: _static/Dropout/Dropout-01.svg :alt: Dropout A *Dropout* layer *k* is an object used for *regularization* of a Neural Network model and aiming at diminishing the *overfitting* of the model to the training data. It takes a *drop_prob* and *axis* argument upon instantiation. The first describes the probability to drop - set to zero - input values forwarded to the next layer *k+1* while the latter selects the axes on which the operation should be applied. .. autoclass:: epynn.dropout.models.Dropout :show-inheritance: Shapes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. automethod:: epynn.dropout.models.Dropout.compute_shapes .. literalinclude:: ./../epynn/dropout/parameters.py :pyobject: dropout_compute_shapes :language: python Within a *Dropout* layer, shapes of interest include: * Input *X* of shape *(m, ...)* with *m* equal to the number of samples. The number of input dimensions is unknown *a priori*. * The number of features *n* per sample can still be determined formally: it is equal to the size of the input *X* divided by the number of samples *m*. * The shape of the dropout mask *D* defined by the user-defined axes *a*. Note that: * The *Dropout* operation is applied on all axis by defaults. Given an input of shape *(m, ...)*, the shape of the dropout mask is therefore identical with default settings. * For input *X* of shape *(m, n)* and *axis=(1)*, the shape of *D* is equal to *(m, 1)*. The *Dropout* operation will set to zero entire columns instead of single values but still with respect to *drop_prob*. Forward ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. automethod:: epynn.dropout.models.Dropout.forward .. literalinclude:: ./../epynn/dropout/forward.py :pyobject: dropout_forward The forward propagation function in a *Dropout* layer *k* includes: * (1): Input *X* in current layer *k* is equal to the output *A* of previous layer *k-1*. * (2): Generate the dropout mask *D* from random uniform distribution. * (3): Evaluate each value in *D* against the dropout probability *d*. Values in *D* lower than *d* are set to *0* while others are set to *1*. * (4): *A* is the product between input *X* and dropout mask *D*. In words, values in *A* are set to *0* with respect to dropout probability *d*. * (5): Values in *A* are scaled-up with respect to fraction of data points that were set to zero by the dropout mask *D*. .. math:: \begin{alignat*}{2} & x^{k}_{mn} &&= a^{\km}_{mn} \tag{1} \\ \\ & Let&&D~be~a~matrix~of~the~same~dimensions~as~x^{k}_{mn}: \\ \\ & D &&= [d_{mn}] \in \mathbb{R}^{M \times N}, \\ & &&\forall_{m,n} \in \{1,...M\} \times \{1,...N\} \\ & && d_{mn} \sim \mathcal{U}[0,1] \tag{2} \\ \\ & D' &&= [d_{mn}'] \in \mathbb{R}^{M \times N}, d \in [0,1] \\ & &&\forall_{m,n} \in \{1,...M\} \times \{1,...N\} \\ & && d_{mn}' = \begin{cases} 1, & d_{mn} > d \\ 0, & d_{mn} \le d \end{cases} \tag{3} \\ \\ & a^{k}_{mn} &&= x^{k}_{mn} * D' \\ & &&~~~* \frac{1}{(1-d)} \tag{4} \end{alignat*} Backward ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. automethod:: epynn.dropout.models.Dropout.backward .. literalinclude:: ./../epynn/dropout/backward.py :pyobject: dropout_backward The backward propagation function in a *Dropout* layer *k* includes: * (1): *dA* the gradient of the loss with respect to the output of forward propagation *A* for current layer *k*. It is equal to the gradient of the loss with respect to input of forward propagation for next layer *k+1*. * (2): The gradient of the loss *dX* with respect to the input of forward propagation *X* for current layer *k* is equal to the product between *dA* and the dropout mask *D*. * (3): Values in *dX* are scaled-up with respect to fraction of data point that were set to zero by the dropout mask *D*. .. math:: \begin{alignat*}{2} & \delta^{\kp}_{mn} &&= \frac{\partial \mathcal{L}}{\partial a^{k}_{mn}} = \frac{\partial \mathcal{L}}{\partial x^{\kp}_{mn}} \tag{1} \\ & \delta^{k}_{mn} &&= \frac{\partial \mathcal{L}}{\partial x^{k}_{mn}} = \frac{\partial \mathcal{L}}{\partial a^{\km}_{mn}} = \frac{\partial \mathcal{L}}{\partial a^{k}_{mn}} * D'^{k}_{mn} * \frac{1}{(1-d)} \tag{2} \\ \end{alignat*} Gradients ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. automethod:: epynn.dropout.models.Dropout.compute_gradients The *Dropout* layer is not a *trainable* layer. It has no *trainable* parameters such as weight *W* or bias *b*. Therefore, there is no parameters gradients to compute. Live examples ------------------------------ * `Protein Modification - Flatten-(Dense)n with Dropout`_ * `Protein Modification - LSTM(sequence=True)-(Dense)n with Dropout`_ * `Dummy time - Flatten-(Dense)n with Dropout`_ * `Author music - Flatten-(Dense)n-with-Dropout`_ * `Author music - RNN(sequences=True)-Flatten-(Dense)n with Dropout`_ * `Author music - GRU(sequences=True)-Flatten-(Dense)n with Dropout`_ * `Dummy image - Flatten-(Dense)n with Dropout`_ * `MNIST Database - Flatten-(Dense)n with Dropout`_ You may also like to browse all `Network training examples`_ provided with EpyNN. .. _Network training examples: run_examples.html .. _Protein Modification - Flatten-(Dense)n with Dropout: epynnlive/ptm_protein/train.html#Flatten-(Dense)n-with-Dropout .. _Protein Modification - LSTM(sequence\=True)-(Dense)n with Dropout: epynnlive/ptm_protein/train.html#LSTM(sequence=True)-(Dense)n-with-Dropout .. _Dummy time - Flatten-(Dense)n with Dropout: epynnlive/dummy_time/train.html#Flatten-(Dense)n-with-Dropout .. _Author music - Flatten-(Dense)n-with-Dropout: epynnlive/author_music/train.html#Flatten-(Dense)n-with-Dropout .. _Author music - RNN(sequences\=True)-Flatten-(Dense)n with Dropout: epynnlive/author_music/train.html#RNN(sequences=True)-Flatten-(Dense)n-with-Dropout .. _Author music - GRU(sequences\=True)-Flatten-(Dense)n with Dropout: epynnlive/author_music/train.html#GRU(sequences=True)-Flatten-(Dense)n-with-Dropout .. _Dummy image - Flatten-(Dense)n with Dropout: epynnlive/dummy_image/train.html#Flatten-(Dense)n-with-Dropout .. _MNIST Database - Flatten-(Dense)n with Dropout: epynnlive/captcha_mnist/train.html#Flatten-(Dense)n-with-Dropout