Flatten - Adapter
Source files in EpyNN/epynn/flatten/
.
See Appendix - Notations for mathematical conventions.
Layer architecture
A Flatten - or reshaping - layer is an object which flattens data arrays from three or more dimensions to two dimensions. It can be seen as an adapter layer because it is necessary, for instance, when the shape of the output of layer k-1 is not compatible with the expected shape of the input for layer k.
- class epynn.flatten.models.Flatten[source]
Bases:
epynn.commons.models.Layer
Definition of a flatten layer prototype.
Shapes
- Flatten.compute_shapes(A)[source]
Wrapper for
epynn.flatten.parameters.flatten_compute_shapes()
.
- Parameters
A (
numpy.ndarray
) – Output of forward propagation from previous layer.def flatten_compute_shapes(layer, A): """Compute forward shapes and dimensions from input for layer. """ X = A # Input of current layer layer.fs['X'] = X.shape # (m, ...) layer.d['m'] = layer.fs['X'][0] # Number of samples (m) layer.d['n'] = X.size // layer.d['m'] # Number of features (n) # Shape for output of forward propagation layer.fs['A'] = (layer.d['m'], layer.d['n']) return NoneWithin a Flatten layer, shapes of interest include:
Input X of shape (m, …) with m equal to the number of samples. The number of input dimensions is unknown a priori.
The number of features n per sample can still be determined formally: it is equal to the size of the input X divided by the number of samples m.
The output shape of the Flatten layer is equal to (m, n).
Forward
- Flatten.forward(A)[source]
Wrapper for
epynn.flatten.forward.flatten_forward()
.
- Parameters
A (
numpy.ndarray
) – Output of forward propagation from previous layer.- Returns
Output of forward propagation for current layer.
- Return type
numpy.ndarray
def flatten_forward(layer,A): """Forward propagate signal to next layer. """ # (1) Initialize cache X = initialize_forward(layer, A) # (2) Reshape (m, ...) -> (m, n) A = layer.fc['A'] = np.reshape(X, layer.fs['A']) return A # To next layerThe forward propagation function in a Flatten layer k includes:
(1): Input X in current layer k is equal to the output A of previous layer k-1.
(2): Output A of current layer k is equal to input X reshaped from (m, …) to (m, n).
Note that:
The reshaping operation preserves the association between samples and corresponding features. The shape (m, …) means there is one row per sample, regardless the number of dimensions within each row. The reshaping operation is applied with respect to each row, therefore preserving data integrity.
\[\begin{split}\begin{alignat*}{2} & x^{k}_{m,d_1...d_n} &&= a^{\km}_{m,d_1...d_n} \tag{1} \\ & a^{k}_{m,n} &&= f(x^{k}_{m,d_1...d_n}) \tag{2} \end{alignat*}\end{split}\]\[\begin{split}\begin{align} where~f~is~defined~as: \\ f:\mathcal{M}_{m,d_1...d_n}(\mathbb{R}) & \to \mathcal{M}_{m,n}(\mathbb{R}) \\ X & \to f(X) \\ with~n \in \mathbb{N}^* \end{align}\end{split}\]
Backward
- Flatten.backward(dX)[source]
Wrapper for
epynn.flatten.backward.flatten_backward()
.
- Parameters
dX (
numpy.ndarray
) – Output of backward propagation from next layer.- Returns
Output of backward propagation for current layer.
- Return type
numpy.ndarray
def flatten_backward(layer, dX): """Backward propagate error gradients to previous layer. """ # (1) dA = initialize_backward(layer, dX) # (2) Reverse reshape (m, n) -> (m, ...) dX = layer.bc['dX'] = np.reshape(dA, layer.fs['X']) return dX # To previous layerThe backward propagation function in a Flatten pass-through layer k includes:
(1): dA the gradient of the loss with respect to the output of forward propagation A for current layer k. It is equal to the gradient of the loss with respect to input of forward propagation for next layer k+1.
(2): The gradient of the loss dX with respect to the input of forward propagation X for current layer k is equal to the reverse of the reshaping operation applied on dA. Therefore, dX has same shape as X which is (m, …).
\[\begin{split}\begin{alignat*}{2} & \delta^{\kp}_{mn} &&= \frac{\partial \mathcal{L}}{\partial a^{k}_{mn}} = \frac{\partial \mathcal{L}}{\partial x^{\kp}_{mn}} \tag{1} \\ & \delta^{k}_{m,d_1...d_n} &&= \frac{\partial \mathcal{L}}{\partial x^{k}_{m,d_1...d_n}} = \frac{\partial \mathcal{L}}{\partial a^{\km}_{m,d_1...d_n}} = f^{-1}(\frac{\partial \mathcal{L}}{\partial a^{k}_{mn}}) \tag{2} \end{alignat*}\end{split}\]
Gradients
- Flatten.compute_gradients()[source]
Wrapper for
epynn.flatten.parameters.flatten_compute_gradients()
. Dummy method, there are no gradients to compute in layer.A Flatten layer is not a trainable layer. It has no trainable parameters such as weight W or bias b. Therefore, there is no parameters to update.
Live examples
Author music - RNN(sequences=True)-Flatten-(Dense)n with Dropout
Author music - GRU(sequences=True)-Flatten-(Dense)n with Dropout
Protein Modification - LSTM(sequence=True)-Flatten-(Dense)n with Dropout
You may also like to browse all Network training examples provided with EpyNN.