Flatten - Adapter

Source files in EpyNN/epynn/flatten/.

See Appendix - Notations for mathematical conventions.

Layer architecture

A Flatten - or reshaping - layer is an object which flattens data arrays from three or more dimensions to two dimensions. It can be seen as an adapter layer because it is necessary, for instance, when the shape of the output of layer k-1 is not compatible with the expected shape of the input for layer k.

class epynn.flatten.models.Flatten[source]

Bases: epynn.commons.models.Layer

Definition of a flatten layer prototype.

Shapes

Flatten.compute_shapes(A)[source]
Wrapper for epynn.flatten.parameters.flatten_compute_shapes().

Parameters

A (numpy.ndarray) – Output of forward propagation from previous layer.
def flatten_compute_shapes(layer, A):
    """Compute forward shapes and dimensions from input for layer.
    """
    X = A    # Input of current layer

    layer.fs['X'] = X.shape    # (m, ...)

    layer.d['m'] = layer.fs['X'][0]          # Number of samples  (m)
    layer.d['n'] = X.size // layer.d['m']    # Number of features (n)

    # Shape for output of forward propagation
    layer.fs['A'] = (layer.d['m'], layer.d['n'])

    return None
Within a Flatten layer, shapes of interest include:

Input X of shape (m, …) with m equal to the number of samples. The number of input dimensions is unknown a priori.

The number of features n per sample can still be determined formally: it is equal to the size of the input X divided by the number of samples m.

The output shape of the Flatten layer is equal to (m, n).

Forward

Flatten.forward(A)[source]
Wrapper for epynn.flatten.forward.flatten_forward().

Parameters

A (numpy.ndarray) – Output of forward propagation from previous layer.

Returns

Output of forward propagation for current layer.

Return type

numpy.ndarray
def flatten_forward(layer,A):
    """Forward propagate signal to next layer.
    """
    # (1) Initialize cache
    X = initialize_forward(layer, A)

    # (2) Reshape  (m, ...) -> (m, n)
    A = layer.fc['A'] = np.reshape(X, layer.fs['A'])

    return A    # To next layer
The forward propagation function in a Flatten layer k includes:

(1): Input X in current layer k is equal to the output A of previous layer k-1.

(2): Output A of current layer k is equal to input X reshaped from (m, …) to (m, n).

Note that:

The reshaping operation preserves the association between samples and corresponding features. The shape (m, …) means there is one row per sample, regardless the number of dimensions within each row. The reshaping operation is applied with respect to each row, therefore preserving data integrity.

\[\begin{split}\begin{alignat*}{2} & x^{k}_{m,d_1...d_n} &&= a^{\km}_{m,d_1...d_n} \tag{1} \\ & a^{k}_{m,n} &&= f(x^{k}_{m,d_1...d_n}) \tag{2} \end{alignat*}\end{split}\]

\[\begin{split}\begin{align} where~f~is~defined~as: \\ f:\mathcal{M}_{m,d_1...d_n}(\mathbb{R}) & \to \mathcal{M}_{m,n}(\mathbb{R}) \\ X & \to f(X) \\ with~n \in \mathbb{N}^* \end{align}\end{split}\]

Backward

Flatten.backward(dX)[source]
Wrapper for epynn.flatten.backward.flatten_backward().

Parameters

dX (numpy.ndarray) – Output of backward propagation from next layer.

Returns

Output of backward propagation for current layer.

Return type

numpy.ndarray
def flatten_backward(layer, dX):
    """Backward propagate error gradients to previous layer.
    """
    # (1)
    dA = initialize_backward(layer, dX)

    # (2) Reverse reshape (m, n) -> (m, ...)
    dX = layer.bc['dX'] = np.reshape(dA, layer.fs['X'])

    return dX    # To previous layer
The backward propagation function in a Flatten pass-through layer k includes:

(1): dA the gradient of the loss with respect to the output of forward propagation A for current layer k. It is equal to the gradient of the loss with respect to input of forward propagation for next layer k+1.

(2): The gradient of the loss dX with respect to the input of forward propagation X for current layer k is equal to the reverse of the reshaping operation applied on dA. Therefore, dX has same shape as X which is (m, …).

\[\begin{split}\begin{alignat*}{2} & \delta^{\kp}_{mn} &&= \frac{\partial \mathcal{L}}{\partial a^{k}_{mn}} = \frac{\partial \mathcal{L}}{\partial x^{\kp}_{mn}} \tag{1} \\ & \delta^{k}_{m,d_1...d_n} &&= \frac{\partial \mathcal{L}}{\partial x^{k}_{m,d_1...d_n}} = \frac{\partial \mathcal{L}}{\partial a^{\km}_{m,d_1...d_n}} = f^{-1}(\frac{\partial \mathcal{L}}{\partial a^{k}_{mn}}) \tag{2} \end{alignat*}\end{split}\]

Gradients

Flatten.compute_gradients()[source]

Wrapper for epynn.flatten.parameters.flatten_compute_gradients(). Dummy method, there are no gradients to compute in layer.

A Flatten layer is not a trainable layer. It has no trainable parameters such as weight W or bias b. Therefore, there is no parameters to update.

Live examples

You may also like to browse all Network training examples provided with EpyNN.