{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Basics with string sequence"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Find this notebook at `EpyNN/epynnlive/dummy_string/train.ipynb`.\n",
"* Regular python code at `EpyNN/epynnlive/dummy_string/train.py`.\n",
"\n",
"Run the notebook online with [Google Colab](https://colab.research.google.com/github/Synthaze/EpyNN/blob/main/epynnlive/dummy_string/train.ipynb).\n",
"\n",
"**Level: Beginner**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this notebook we will review:\n",
"\n",
"* Handling sequential string data.\n",
"* Training of Feed-Forward (FF) and recurrent networks (RNN, LSTM, GRU).\n",
"* Differences between decisions and probabilities and related functions.\n",
"\n",
"**It is assumed that all *basics* notebooks were already reviewed:**\n",
"\n",
"* [Basics with Perceptron (P)](../dummy_boolean/train.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**This notebook does not enhance, extend or replace EpyNN's documentation.**\n",
"\n",
"**Relevant documentation pages for the current notebook:**\n",
"\n",
"* [Fully Connected (Dense)](https://epynn.net/Dense.html)\n",
"* [Recurrent Neural Network (RNN)](https://epynn.net/RNN.html)\n",
"* [Long Short-Term Memory (LSTM)](https://epynn.net/LSTM.html)\n",
"* [Gated Recurrent Unit (GRU)](https://epynn.net/GRU.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Environment and data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Follow [this link](prepare_dataset.ipynb) for details about data preparation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Briefly, these dummy string data consist of sequences of characters. Sample features are each represented by one sequence and can be either associated with a positive or negative label.\n",
"\n",
"Positive sequences are met when the first element in the sequence is equal to the last element in this same sequence, and reciprocally."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# EpyNN/epynnlive/dummy_string/train.ipynb\n",
"# Install dependencies\n",
"!pip3 install --upgrade-strategy only-if-needed epynn\n",
"\n",
"# Standard library imports\n",
"import random\n",
"\n",
"# Related third party imports\n",
"import numpy as np\n",
"\n",
"# Local application/library specific imports\n",
"import epynn.initialize\n",
"from epynn.commons.io import one_hot_decode_sequence\n",
"from epynn.commons.maths import relu, softmax\n",
"from epynn.commons.library import (\n",
" configure_directory,\n",
" read_model,\n",
")\n",
"from epynn.network.models import EpyNN\n",
"from epynn.embedding.models import Embedding\n",
"from epynn.flatten.models import Flatten\n",
"from epynn.rnn.models import RNN\n",
"from epynn.gru.models import GRU\n",
"from epynn.lstm.models import LSTM\n",
"from epynn.dense.models import Dense\n",
"from epynnlive.dummy_string.prepare_dataset import prepare_dataset\n",
"from epynnlive.dummy_string.settings import se_hPars\n",
"\n",
"\n",
"########################## CONFIGURE ##########################\n",
"random.seed(1)\n",
"\n",
"np.set_printoptions(threshold=10)\n",
"\n",
"np.seterr(all='warn')\n",
"\n",
"configure_directory()\n",
"\n",
"\n",
"############################ DATASET ##########################\n",
"X_features, Y_label = prepare_dataset(N_SAMPLES=480)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's control what we retrieved."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"480\n",
"12\n",
"['G', 'A', 'C', 'T', 'T', 'G', 'G', 'C', 'C', 'A', 'T', 'C']\n",
"1\n"
]
}
],
"source": [
"print(len(X_features))\n",
"print(len(X_features[0]))\n",
"print(X_features[0])\n",
"print(Y_label[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We retrieved a set of sample features describing ``480`` samples.\n",
"\n",
"Each sample is described by ``12`` string features.\n",
"\n",
"Herein the label is ``1`` because the first and last element are different."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Feed-Forward (FF)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To compare Feed-Forward and recurrent networks, we are going to train a simple Perceptron first."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Embedding"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The principle of [One-hot encoding of string features](prepare_dataset.ipynb#One-hot-encoding-of-string-features) was detailed before.\n",
"\n",
"Briefly, we can not do math on string data. Therefore, the one-hot encoding process may be summarized as such:\n",
"\n",
"* List of all elements of size vocab_size. This basically answers: what is the number of distinct elements we can find in your data?\n",
"* Each element is associated with one index in the range(0, vocab_size). This provides an ``element_to_idx`` encoder.\n",
"* For one sample and for each element in the associated list of features, a zero array is initialized. This array is set to one at the index which is assigned to the ``element_to_idx`` encoder.\n",
"\n",
"This is achieved during instantiation of the *embedding* layer by setting up ``X_encode=True``."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"embedding = Embedding(X_data=X_features,\n",
" Y_data=Y_label,\n",
" X_encode=True,\n",
" Y_encode=True,\n",
" relative_size=(2, 1, 0))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's inspect some properties."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'A': 0, 'C': 1, 'G': 2, 'T': 3}\n",
"[[0. 0. 1. 0.]\n",
" [1. 0. 0. 0.]\n",
" [0. 1. 0. 0.]\n",
" ...\n",
" [1. 0. 0. 0.]\n",
" [0. 0. 0. 1.]\n",
" [0. 1. 0. 0.]]\n",
"{0: 'A', 1: 'C', 2: 'G', 3: 'T'}\n",
"['G', 'A', 'C', 'T', 'T', 'G', 'G', 'C', 'C', 'A', 'T', 'C']\n"
]
}
],
"source": [
"print(embedding.e2i) # element_to_idx\n",
"print(embedding.dtrain.X[0])\n",
"\n",
"print(embedding.i2e) # idx_to_element\n",
"print(one_hot_decode_sequence(embedding.dtrain.X[0], embedding.i2e))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Encoded sequences may be decoded as shown above."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Flatten-Dense - Perceptron"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's inspect the shape of the data."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(320, 12, 4)\n"
]
}
],
"source": [
"print(embedding.dtrain.X.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It contains 320 samples (m), each described by a sequence of 12 features (s) containing 4 elements (v).\n",
"\n",
"12 features is the length of the sequences and 4 elements is the size of the vocabulary. Remember one-hot encoding makes a zero array of this size and sets 1 at the index corresponding to the element being encoded.\n",
"\n",
"Still, the fully-connected or *dense* layer can only process bi-dimensional input arrays. That is the reason why we need to invoke a *flatten* layer in between the *embedding* and *dense* layer."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"name = 'Flatten_Dense-2-softmax'\n",
"\n",
"se_hPars['learning_rate'] = 0.001\n",
"\n",
"flatten = Flatten()\n",
"\n",
"dense = Dense(2, softmax)\n",
"\n",
"layers = [embedding, flatten, dense]\n",
"\n",
"model = EpyNN(layers=layers, name=name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Initialize using most classically a *MSE* or *Binary Cross Entropy* loss function."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m--- EpyNN Check OK! --- \u001b[0m\r"
]
}
],
"source": [
"model.initialize(loss='BCE', seed=1, se_hPars=se_hPars.copy(), end='\\r')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Train for hundred epochs."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m\u001b[37mEpoch 49 - Batch 0/0 - Accuracy: 0.738 Cost: 0.55582 - TIME: 2.14s RATE: 2.34e+01e/s TTC: 0s \u001b[0m\n",
"\n",
"+-------+----------+----------+-------+--------+-------+------------------------------------+\n",
"| \u001b[1m\u001b[37mepoch\u001b[0m | \u001b[1m\u001b[37mlrate\u001b[0m | \u001b[1m\u001b[32maccuracy\u001b[0m | | \u001b[1m\u001b[31mBCE\u001b[0m | | \u001b[37mExperiment\u001b[0m |\n",
"| | \u001b[37mDense\u001b[0m | \u001b[1m\u001b[32mdtrain\u001b[0m | \u001b[1m\u001b[32mdval\u001b[0m | \u001b[1m\u001b[31mdtrain\u001b[0m | \u001b[1m\u001b[31mdval\u001b[0m | |\n",
"+-------+----------+----------+-------+--------+-------+------------------------------------+\n",
"| \u001b[1m\u001b[37m0\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.688\u001b[0m | \u001b[1m\u001b[32m0.694\u001b[0m | \u001b[1m\u001b[31m0.631\u001b[0m | \u001b[1m\u001b[31m0.640\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m5\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.716\u001b[0m | \u001b[1m\u001b[32m0.731\u001b[0m | \u001b[1m\u001b[31m0.609\u001b[0m | \u001b[1m\u001b[31m0.620\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m10\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.719\u001b[0m | \u001b[1m\u001b[32m0.750\u001b[0m | \u001b[1m\u001b[31m0.596\u001b[0m | \u001b[1m\u001b[31m0.609\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m15\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.728\u001b[0m | \u001b[1m\u001b[32m0.750\u001b[0m | \u001b[1m\u001b[31m0.585\u001b[0m | \u001b[1m\u001b[31m0.601\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m20\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.728\u001b[0m | \u001b[1m\u001b[32m0.750\u001b[0m | \u001b[1m\u001b[31m0.577\u001b[0m | \u001b[1m\u001b[31m0.595\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m25\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.756\u001b[0m | \u001b[1m\u001b[31m0.571\u001b[0m | \u001b[1m\u001b[31m0.590\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m30\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.756\u001b[0m | \u001b[1m\u001b[31m0.566\u001b[0m | \u001b[1m\u001b[31m0.587\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m35\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.562\u001b[0m | \u001b[1m\u001b[31m0.585\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m40\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.559\u001b[0m | \u001b[1m\u001b[31m0.583\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m45\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.756\u001b[0m | \u001b[1m\u001b[31m0.557\u001b[0m | \u001b[1m\u001b[31m0.582\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m49\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.741\u001b[0m | \u001b[1m\u001b[32m0.756\u001b[0m | \u001b[1m\u001b[31m0.555\u001b[0m | \u001b[1m\u001b[31m0.581\u001b[0m | \u001b[37m1635014418_Flatten_Dense-2-softmax\u001b[0m |\n",
"+-------+----------+----------+-------+--------+-------+------------------------------------+\n"
]
}
],
"source": [
"model.train(epochs=50, init_logs=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot the results."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABKiklEQVR4nO3dd3hUZdr48e+dZNJ7o0NCLwmhg9gQRLFhL6x9V3l1V11ddXXV18X6c1d9d93VXVcs6Fqwu8paERDphCIQuhAgQRII6XUy8/z+OCdxCJOQQJJJuT/Xda6Z85x2n8lk7vOc55zniDEGpZRSqi4/XweglFKqbdIEoZRSyitNEEoppbzSBKGUUsorTRBKKaW80gShlFLKK00QSimlvNIEoTolEZkkIlm+jkM1jYgMEpH1IlIsInf4Op6OThNEGyYit4lIuohUisicOtNCReQfInJIRApFZLHHtLtEZJeIFInIfhH5i4gEeEzPFJFyESmxh6/rrPsuETlgL/+qiAR5THtMRDaKSLWIzGog9ldFxIhIfy/TBohIhYi86VHWTUQ+teM1IpJUZ5lYEXlXRPLsfX5LRCIb8RkaESn12NeCYy3jZR03iMiSOmVzROTxpq7rOLZd87cqFpECEVkmIreISJv63xWR60Vkjf2dyRKRP3t+55rR74GFxpgIY8zf7M/nzBbYjkITRFu3H3gceNXLtJeAWGCI/XqXx7RPgVHGmEggBUgD6h5tXWCMCbeHs2oKReRs4H5gCtAH6As84rHcTqx/0v/WF7SInAL0a2C/XgBW1ylzA18Cl9azzONADJBsr7sLMKuBbXhK89jX6EYu05ZcYIyJwPp7PAXcB7zi25COEgrcCcQD47G+P/e0wHb6ABktsF7ljTFGhzY+YP04zvEYHwwUAZGNWDYOmA/8w6MsEziznvnfBp70GJ8CHPAy35vALC/lAcA6YDhggP51pl8FvIf14/5mPcsbIKlO+RfArz3GfwN81Yj9PyoGu3wSkOUxfj/wI1AMbAYutsuHABWACygBCoCZgBOosss+s+ftDnwIHAR2A3d4rH+Wvd9v2NvIAMY0Iv6j/lbAOKyEmmKPBwHPAHuBHOBFIMRzP4G7gVzgJ+BGj3Wda+9vMZAN3OMx7Xxgvb3Py4DhTfjO/q7mc6ln+jgg3f4e5wD/5zFtuv35FACLgCF2+QL771Bhf+7v2J9DuT3+eyDJ/pvfCOwD8oFbgLHABnudz3tsq5+93jzgEPAWEO0x7TDWwVbN3/cgMMmXvwetOfg8AB0a8Uc6OkFcB2wE/mJ/qTcCl9ZZ5hf2P5+xv9RpHtMy7X/Kg8DXdab9AFzpMR5vryOuzvrrSxD3As/Z74/4cQYige1AT5qeIM4HPseqRcTY/9R3NuKza2yCuNz+AfADrgRKgW72tBuAJXWWnwM87jHuB6wBHgYCsWpeu4Cz7emz7B+2cwF/4P8BKxoRfyZekjlWMrjVfv8XrFpjLBABfAb8P4/9rAYeBRz29suAGHv6T8Cp9vsYjx/DkVgJZbwd7/V2LEGN/M5+AjzVwPTlwLX2+3Bggv1+oP3ZT7Xj/T1WrTXQnr4IuKm+z4efE8SLQDBwlv25fwIkAj3s/Trdnr+/va0gIAFYDPzVY303YyXQUOAr4Blf/x605qCnmNqnnlinjgqxftRuA14XkSE1Mxhj3jbWKaaBWP8sOR7LX431j9QHWAh8JSLR9rRwe701at5HHCsoEekF/A/Wj6Q3jwGvGGOOp3F4LdYPb549uIB/NHZZ+/x9gYj8zdsMxpj3jTH7jTFuY8y7wA6so9zGGgskGGMeNcZUGWN2AbOxakw1lhhjPjfGuIB/Y536O177gVgREawazV3GmMPGmGLgyTrbdQKPGmOcxpjPsY62B3lMGyoikcaYfGPMWrt8JvAvY8xKY4zLGPM6UAlMOFZgIvJLYAxWraY+TqC/iMQbY0qMMSvs8iuB/xpjvjHGOO11hAATj7XdOh4zxlQYY77GSjjvGGNyjTHZwPdYCRBjzE57W5XGmIPA/wGn16zEGDMbK0GtBLoBDzYxjnZNE0T7VI71D/a4/WP0HdYP/Vl1ZzTG7MCqrv/Do2ypMabcGFNmjPl/WNXuU+3JJVhH+jVq3hc3Iq6/Yv0QFdadICIjgDOxjnaPx3tYtY8IO6YfsWoxjTHKGBNtD16vfBGR6+yrYwrshuwUrNpTY/UBunskogLgAay2khoHPN6XAcEn0JDbA+v0RwLW0e0aj+1+aZfXyDPGVNfZdrj9/lKsWsUeEflORE7y2J+76+xPL3sfr/Zo9P/CMygRuQirdnSOMeaQXeZt/l9hHbxsFZHVInK+Xd4d2FOzPmOMG+tUUY8mfj6eB0TlXsbD7di6iMhcEckWkSKs71Tdv/tsrO/D340xlU2Mo11riasMVMvb4KWsoX7bA2i40dgAYr/PwDqyfc8eTwNyjDF5jYhrCnCKiPzZo2y5iPwWq3qfBOy1DnoJB/xFZKgxZlQj1j0C+I0xphRARF4EljS4RCOJSB+sH4EpwHJjjEtE1vPzZ+Lts61btg/YbYwZ0BwxNURExmL9YC7BOsVYDgyzj46bxBizGrhQRBxYNdH3sBLBPuAJY8wT9Sz6lpe4pmF9jucZYzZ6bOOtuvPbBy4z7KuxLgE+EJE4rJpRqsc6xY6nvn070ecVPGmvI9UYc9hOcM97bD8c68DnFWCWiHxojDl8gttsN7QG0YaJSICIBGOdA/YXkZojzsVY56D/YM9zMnAG1jlSROQmEUm03w8F/gB8a4/3FpGTRSTQXt+9WEdMS+3NvgH8SkSG2qedHsI6314Tk8OOyQ8IsNfhb08eiJVQRtgDwAXAx1hXXfXzmPYi1pVQZ3usOxjrXDBAkD1eYzVwk4iEiEgI1ikQb4nyeITxc1sNInIj1hFjjRygp4gE1inr6zG+CigWkfvsGP1FJMX+MW8WIhJpH2nPxWq/2WgfYc8G/uLxN+9hX412rPUF2kf3UfbpnCKsRl/sdd4iIuPFEiYi54mI11ONIjIZKwlcaoxZ1YhtXyMiCXb8BXaxGytBnSciU+ykdTfWqa1l9ayq7t+hqSKwas2FItIDqw3N03NAujHmJqzv64snsK32x9eNIDrUP2A1bJo6wyx72jCshr5SPK66sae9hvWPU4rViPc0EOyx3AZ7Wh5W4hhTZ7u/s5cvstcV5DFtjpeYbqgnfq8NxB779qaX+Y8YPKYlYzW+5mGdWvkSGNCIz7CxjdRP2Os9hHUe+jvsxlCsto//1ky3ywbw8xU+n9hl3bGurDmAdfXMCuwG1Lr7y8+NqQHHiD8Tq4ZQjNUetBzrCi5/j3mCsY6Ed9l/sy3YV1DV3U+PdZ5p79eXdqxFWEn4FI/5ptllBViN2e8DEfXEuRCrMbzEY/iigf16E6uxuASr1nqRx7SLsb7ThfbfYZjHtEUc2Uh9IdbBUgHWZbVHfa5YV3FNqrPthzz+H9bYcazHSkhZHuvOBmLt8XCs9oirff3b0FqD2DuulFJKHUFPMSmllPJKG6lVuyYip2LdRHcUY0y4t/K2RER6Y51O8WaoMWZva8ajlCc9xaSUUsqrDlODiI+PN0lJSb4OQyml2pU1a9YcMsYkeJvWYRJEUlIS6enpvg5DKaXaFRHZU980baRWSinllSYIpZRSXmmCUEop5ZUmCKWUUl5pglBKKeWVJgillFJeaYJQSinlVYe5D0Ip1QZkfAI5Gb6Owjf6ToKkk30dRbPSBKGUOnHGwHd/gkX/zy6QBmfveAx8/yxcOhtSLvV1MM1GE4RS6sQYA/NnwdK/woirYfrfwc//WEt1LJXF8PaV8OFN4KyAkVf7OqJmoW0QSqnj53bDF/dZyWHMr2D6850vOQAERcDVH0Dy6fCfX8Oq2b6OqFloglBKHR+3C+b9Flb9C066Dc57Fvw68U9KYCjMmAsDz4HP74Flf/d1RCesE/81lVLHzVUNH/8PrH0DTvs9nPU4SGdrd/DCEQxX/huGXgRfPwTf/dk6BddOaRuEUjX2roSt83wdRftwYAPsWgRTHoZT7/Z1NG2LvwMufQUcIbDwCcjdAlE9W3abUb1g/MxmX60mCKUAtn4O719vHe356b/FMfkHwrQ/wYRbfB1J2+QfABf+A4KjYe3rLV+L6DFKE4RSLWLTh/DRTOiWBtd8CCExvo5IdQR+fnDOU9bQTmkbhOrc1r9tXZrYcxxc+4kmB6U8tGiCEJFpIrJNRHaKyP1epv9FRNbbw3YRKfCY5vKY9mlLxqk6qdWvwCe3WpcmXvMhBEf6OiKl2pQWO8UkIv7AC8BUIAtYLSKfGmM218xjjLnLY/7bgZEeqyg3xoxoqfhUJ7f8BfjqAeuSxMvnWFefKKWO0JJtEOOAncaYXQAiMhe4ENhcz/wzgD+2YDwdW9F+60fP5fR1JG1f2SGr3WHoRXDJbAgI9HVESrVJLZkgegD7PMazgPHeZhSRPkAysMCjOFhE0oFq4CljzCdelpsJzATo3bt380TdXi1/wRqCo3wdSfsw+kY49xnrahOllFdt5b/jKuADY4zLo6yPMSZbRPoCC0RkozHmR8+FjDEvAS8BjBkzpv3ejXKi3G7I+BgGToNfzPV1NEqpDqIlG6mzgV4e4z3tMm+uAt7xLDDGZNuvu4BFHNk+oTztXQ5F2ZB6ma8jUUp1IC2ZIFYDA0QkWUQCsZLAUVcjichgIAZY7lEWIyJB9vt44GTqb7tQmz6AgBCrBqGUUs2kxU4xGWOqReQ24CvAH3jVGJMhIo8C6caYmmRxFTDXmCNuNRwC/EtE3FhJ7CnPq5+UB5fTekjLoHMgKNzX0SilOpAWbYMwxnwOfF6n7OE647O8LLcMSG3J2DqMXYug/LCeXlJKNTu9k7q92/ShdeVS/zN9HYlSqoPRBNGeOcthyzwYcgEEBPk6GqVUB9NWLnNVx2PH11BVDCl6ekmpzqi0spq3Vu6hqtrNbZMHNPv6NUG0Zxs/gLBESD7N15EopVpRYZmTOcsyeW3ZbgrKnJw5pAvGGKSZH9qkCaK9qiiC7V/B6Os75zOAleqEDhZX8sqS3by5Yg8lldWcOSSRX5/Rn1G9W6YXYk0Q7dXW/4KrUk8vtTC32/BlxgF+yCrgslE9GdAlwtchqQ5sU3Yhc1fvpbTSddS0ymoX327Jpcrl5rzUbvzmjP4M6dayPRBrgmivNn0AUb2h1zhfR9IhOV1uPl2/n38s2smPB0sBeGnxLqYN68pvzuhPSg/t80o1n/TMwzy/cCeLth0kNNCf+HDvF51MT+vOrZP60Tehde550gTRHpXmwY8LYeLt+qD4ZlbhdPHBmixe/O5HsvLLGdw1gr/PGMmEvnG8sTyTOcsy+WLTAU4fmMBtk/szNinW1yGrdsoYw5Kdh3h+wU5W7j5MbFgg9549iGtP6kNksMPX4QGaINqnzZ+AcenNcc3ssx/289i8zeQWVzKiVzSzLhjGlCGJtQ1/d581iJtP68u/l+/hlSW7ufzF5YzqHU2PmFAfR96+xYY6uPakPvRPPP7Td7nFFbyxbA97Dpc1Y2QtK/NQKRuzC+kSGcT/nj+UGeN6ERrYtn6SxbT0w7RbyZgxY0x6erqvw2gdr50LpYfgNyu1BtFM3lq5hwc/3kRar2h+f/YgJvaLa/CKkPIqF++s2sv7a7KodB59vlg13v7Cciqr3cd1+i67oJx/ffcj767eh9Plpk9cGO3lPyIsKIAZ43pz6egeBAX47kITEVljjBnjbVrbSlfq2AqzYc8yOOMBTQ7N5JUlu3ls3mYmD07kH1ePIthx7H/WkEB/fnlKMr88JbkVIuzYDpdW8eqS3bxun76bNCiB287oz5gGTt/tOljCPxf9yMfrshGBS0b25NZJ/UiKD2vFyDs+rUG0hrLD8NWD4Cw98XUVZkN2Oty+FuL6nfj6OqiM/YW8n57F8J5RTE/rToC/904Dnl+wg2e+3s65qV3565UjCQzQzgV8pajCWXv67nBpFSN7R9Mt6uhHwRZXVLN05yEc/n7MGNebmaf1pXt0iA8i7hgaqkFogmgNXz4AK/8J8QObZ309x8CFLzTPujqYNXvyeWHhThZszcVPwG2gV2wIt5zej8tG96ytyhtjeObrbbyw8EcuHtmDpy8bXm8SUa2rrKqad1bt4+N1WVQ63UdN9xPhjMGJ/OqUZBIitIuZE6UJwpcKs+FvI60G5Yv+4etoOiRjDMt+zOP5BTtZviuPmFAHvzw5metOSmKVffngD/sK6BIZxM2n9mXGuN48+/V2Xl26mxnjevHERan4+enpOtU5aYLwpc/uhHVvwu1rIKaPr6NpEzIPlTJnWSb7C8qbZX37C8vZlF1EYkQQM0+zEkBY0M/Na8YYlu7M4/mFO1ix6zDBDj8qnG5umJjEHy8Y2uzdEyjVnmgjta8c3gXr/g2jb9DkAGw7UMw/Fu3ksx/2E+DvR99malAMdvjz+EUpXDa6p9cGZhHhlAHxnDIgnjV7DjN78W6Gdo/k9sn9NTko1QBNEC1p0VPg54DT7vV1JD61IauA5xfs5OvNOYQG+nPTqX256ZRkEiOPboBsaaP7xDL6Wr25TanG0ARxPHbMh5yNMPG34FdPw2buFtjwnnW3c0TX1o2vhczfnMPc1ftoymnJ/LIq1u4tIDI4gDumDODGiUnEhAW2YJRKqeaiCeJ4LPkL7FkCuVutq4n8vXyMC5+AwHA45a7Wj68FfLQ2i3ve/4GukcHEhjf+B97fz4/7pg3mmgm9iWgj3QcopRpHE0RTGQO5GRDZAzbMheoKuGQ2BHj8aO5fB1s+g9Pvh9D2fzrj7ZV7efCTjZzUN46Xrx/T5roDUEq1DP1Pb6qSHCjPh0l/AJcTvn7QShKXv87SPSV8uCaLPxb9kaiQGDjpN76O9oTV3GV8xqAE/nnN6EbdZayU6hg0QTRVTob1mjjEepKbIxj+ezd5L1/Mr7NmMsC9i6ig73gz4lf02VfJKf2b/ylPreWFhTt5+qttTBvWlb/N0LuMlepsNEE0Ve4W6zVxqPU69ibWH6gkNf1B3g4pZEBCKGV58bxUMYW9r6wirVc0t53RnymDE9vNzVjGGJ79ejvPL9zJhSO68+zlaXqXsVKdkCaIpsrdYj0HOiwegA/XZHHvsiRuT7yPO4ueRn5yEXjuM3wzalrtcwVufiOdwV0j+PUZ/TkvtRv+zZwoap5h8FXGAdzNcONjaaWL9fsKuGpsL564OLXZ41VKtQ96J3VTvTQJgiLh+k9ru4g+uX8cs68bQ2jmt1bj9HnPQoDVR0y1y81nG/bzwsIf2ZlbQnJ8GLee3o+LRvY44VM2ZVXVvL1yLy8t3kVucSX9E8OJDmmeK4VO7h/Pb6cMaDe1HqXU8dGuNpqL2w1PdocxN/JK+MwmdRHtdhu+yjjA8wt3krG/iO5Rwcw8rS9Xjevd5IbfwnInbyzL5NWlu8kvc3JS3zhum9z/mM8wUEqpurSrjeaSvxuqy1lekshjizZzTkpXnruqcY23fn7COandmJbSlUXbD/LCgp3M+mwzzy/cSb8mPF/WAFv2F1FcWc3kwYn85oz+jO4TcwI7pZRS3mmCaAq7gfqV7SGMTYrh7zNGNrnxVkQ4Y1AiZwxKZOWuPOYsy+RwaVXjlwemDu3Cr05NZlj3xj95SymlmkoTRFPYCWJZUQKPnd37hK/sGd83jvF945ojMqWUanZ67WJT5GZwOLA7roBQzhrWxdfRKKVUi9IE0QQmZwsbnd2ZPDhR+xVSSnV4miAaq7oS8naw0dmD6WndfR2NUkq1OE0QjXVoB2Jc7PFP4ozBib6ORimlWpwmiEZyHrD6YEroO0I7rFNKdQqaIBpp39Z0qow/48aO83UoSinVKjRBNFLpvo3ske6cPEjbH5RSnYMmiEYorawmtmQHZdGDcGivpkqpTkJ/7Rph0YYf6SGHiE1O83UoSinVajRBNML6NSsA6DFwtI8jUUqp1tOiCUJEponINhHZKSL3e5n+FxFZbw/bRaTAY9r1IrLDHq5vyTgbUlBWRVnWBgD8ug7zVRhKKdXqWqwvJhHxB14ApgJZwGoR+dQYs7lmHmPMXR7z3w6MtN/HAn8ExmB1YLrGXja/peKtz5ebDtCffbgCQvGP6t3am1dKKZ9pyRrEOGCnMWaXMaYKmAtc2MD8M4B37PdnA98YYw7bSeEbYFoLxlqvT3/YT1rgfvy6DAU/PSOnlOo8WvIXrwewz2M8yy47ioj0AZKBBU1ZVkRmiki6iKQfPHiwWYL2lFtUwfJdeQz2y0IShzT7+pVSqi1rK4fEVwEfGGNcTVnIGPOSMWaMMWZMQkJCswc1b8NPxJpCQqvzoYu2PyilOpeWTBDZQC+P8Z52mTdX8fPppaYu2yJcbsOHa7OYGpdnFWgNQinVybRkglgNDBCRZBEJxEoCn9adSUQGAzHAco/ir4CzRCRGRGKAs+yyVvPKkl1k7C9iRlKJVZCoNQilVOfSYlcxGWOqReQ2rB92f+BVY0yGiDwKpBtjapLFVcBcY4zxWPawiDyGlWQAHjXGHG6pWHFWgCO4dnTrgSKe+Wo7Zw3twvDAbAiNh/DmP4WllFJtmXj8LrdrY8aMMenp6U1fsDALXjsHJj0AI2ZQWe3iwueXcqikkq/uPI24d84FRwjcMK/5g1ZKKR8TkTXGmDHepukzqUNiICYZPrkFqsv5v4MnsfVAMa9cP4a4UAcc3Aojr/F1lEop1eraylVMvhMYBr94DwacDfPuwrn0BWaM68WUIV2gcC9UlWgDtVKqU9IaBIAjmOKLXiP92Ut5OODfVMX0BoZD7hZrujZQq07O6XSSlZVFRUWFr0NRxyk4OJiePXvicDgavYwmCNtjX+zk4/Jfs2JIInHfPQHuSqvtASBxsG+DU8rHsrKyiIiIICkpCRHxdTiqiYwx5OXlkZWVRXJycqOX0wQBfJ1xgPfSs/jNGQOJm/oazLsTvn8GgqMgujcERfg6RKV8qqKiQpNDOyYixMXF0dQeJzp9G8TB4kr+8NFGhnWP5LdTBoKfP5z/HIz7H6gohARtf1AK0OTQzh3P36/T1yAC/f04fVACt57ej8AAO1/6+cE5f4KuqdrFhlJt0KxZswgPD+eee+5hzpw5nHXWWXTv3rTHAb/44ouEhoZy3XXXtVCU7V+nTxBRoQ7+74oRR08QgVHXtno8SqmmmTNnDikpKV4ThMvlwt/f3+tyt9xyS0uHdtyqq6sJCPD9z3OnP8WklGofnnjiCQYOHMgpp5zCtm3bAPjggw9IT0/n6quvZsSIEZSXl5OUlMR9993HqFGjeP/995k9ezZjx44lLS2NSy+9lLKyMsCqhTzzzDMATJo0ifvuu49x48YxcOBAvv/++6O2X1JSwpQpUxg1ahSpqan85z//qZ32xhtvMHz4cNLS0rj2WuvAMicnh4svvpi0tDTS0tJYtmwZmZmZpKSk1C73zDPPMGvWrNoY7rzzTsaMGcNzzz3HZ599xvjx4xk5ciRnnnkmOTk5tXHceOONpKamMnz4cD788ENeffVV7rzzztr1zp49m7vuqn3cznHzfYpSSrUrj3yWweb9Rc26zqHdI/njBfWfzl2zZg1z585l/fr1VFdXM2rUKEaPHs1ll13G888/zzPPPMOYMT/fDBwXF8fatWsByMvL4+abbwbgoYce4pVXXuH2228/ahvV1dWsWrWKzz//nEceeYT58+cfMT04OJiPP/6YyMhIDh06xIQJE5g+fTqbN2/m8ccfZ9myZcTHx3P4sNUr0B133MHpp5/Oxx9/jMvloqSkhPz8hp95VlVVRU2PEPn5+axYsQIR4eWXX+bPf/4zzz77LI899hhRUVFs3Lixdj6Hw8ETTzzB008/jcPh4LXXXuNf//rXsT72Y9IEoZRq877//nsuvvhiQkNDAZg+fXqD81955ZW17zdt2sRDDz1EQUEBJSUlnH322V6XueSSSwAYPXo0mZmZR003xvDAAw+wePFi/Pz8yM7OJicnhwULFnD55ZcTHx8PQGxsLAALFizgjTfeAMDf35+oqKhjJgjPuLOysrjyyiv56aefqKqqqr08df78+cydO7d2vpiYGAAmT57MvHnzGDJkCE6nk9TU1Aa31RiaIJRSTdLQkX5bERYWVvv+hhtu4JNPPiEtLY05c+awaNEir8sEBQUB1o95dXX1UdPfeustDh48yJo1a3A4HCQlJTX5xsGAgADcbnfteN3lPeO+/fbb+d3vfsf06dNZtGhR7amo+tx00008+eSTDB48mBtvvLFJcdVH2yCUUm3eaaedxieffEJ5eTnFxcV89tlntdMiIiIoLi6ud9ni4mK6deuG0+nkrbfeOu4YCgsLSUxMxOFwsHDhQvbs2QNYR+7vv/8+eXnWs2NqTjFNmTKFf/7zn4DVWF5YWEiXLl3Izc0lLy+PyspK5s2rvxPQwsJCevSwHqT5+uuv15ZPnTqVF154oXa8plYyfvx49u3bx9tvv82MGTOOez89aYJQSrV5o0aN4sorryQtLY1zzjmHsWPH1k674YYbuOWWW2obqet67LHHGD9+PCeffDKDBx9/rwhXX3016enppKam8sYbb9Sua9iwYTz44IOcfvrppKWl8bvf/Q6A5557joULF5Kamsro0aPZvHkzDoeDhx9+mHHjxjF16tQG45k1axaXX345o0ePrj19BVY7Sn5+PikpKaSlpbFw4cLaaVdccQUnn3xy7WmnE6XdfSuljmnLli0MGaI3jbZ1559/PnfddRdTpkzxOt3b37Gh7r61BqGUUu1cQUEBAwcOJCQkpN7kcDy0kVoppdq56Ohotm/f3uzr1RqEUkoprzRBKKWU8koThFJKKa80QSillPJKE4RSqt3x7GivqW644QY++OCDZo6oY2p0ghCR0JYMRCmlOgNv3Xi0VcdMECIyUUQ2A1vt8TQR+UeLR6aUUh68dfe9detWxo0bVztPZmZmbSd1jz76KGPHjiUlJYWZM2dyrJuC6+sW3Fu33eC9i++6tZPw8HAAFi1axKmnnsr06dMZOnQoABdddBGjR49m2LBhvPTSS7XLfPnll4waNYq0tDSmTJmC2+1mwIABtY8Ldbvd9O/fv8mPDz0ejbkP4i/A2cCnAMaYH0TktBaNSinVdn1xPxzY2Lzr7JoK5zxV7+T6uvsePHgwVVVV7N69m+TkZN59993aHlFvu+02Hn74YQCuvfZa5s2bxwUXXFDvNi655BKv3YJ767Y7IyPDaxffDVm7di2bNm2q7ZX11VdfJTY2lvLycsaOHcull16K2+3m5ptvZvHixSQnJ3P48GH8/Py45ppreOutt7jzzjuZP38+aWlpJCQkNPrjPV6NOsVkjNlXp8jVArEopZRXnt19R0ZGHtHd9xVXXMG7774LcESCWLhwIePHjyc1NZUFCxaQkZHR4DY2bdrEqaeeSmpqKm+99Vbt/AsWLODWW28Ffu62u74uvhsybty42uQA8Le//Y20tDQmTJjAvn372LFjBytWrOC0006rna9mvb/85S9ruw5/9dVXm6231mNpTA1in4hMBIyIOIDfAltaNiylVJvVwJG+L1x55ZVcfvnlXHLJJYgIAwYMoKKigl//+tekp6fTq1cvZs2adcyuuRvbLXhDPLvzdrvdVFVV1U7z7Mp70aJFzJ8/n+XLlxMaGsqkSZMajK9Xr1506dKFBQsWsGrVqhPqlbYpGlODuAX4DdADyAZG2ONKKdUqGuruu1+/fvj7+/PYY4/V1h5qfmzj4+MpKSlp1FVL9XUL7q3b7vq6+E5KSmLNmjUAfPrppzidTq/bKiwsJCYmhtDQULZu3cqKFSsAmDBhAosXL2b37t1HrBes5z1cc801XH755fU+Z7u5HTNBGGMOGWOuNsZ0McYkGmOuMcbktUZwSikFDXf3DVYt4s033+SKK64ArL6Jbr75ZlJSUjj77LOPmt+b+roF99Ztd31dfN9888189913pKWlsXz58iNqDZ6mTZtGdXU1Q4YM4f7772fChAkAJCQk8NJLL3HJJZeQlpZ2xBPmpk+fXvs86tZyzO6+ReQ14KiZjDG/bKmgjod2961Uy9Huvn0vPT2du+66i++///6419HU7r4b0wbh+cijYOBiYP9xR6iUUqpJnnrqKf75z3+2WttDjWMmCGPMh57jIvIOsKTFIlJKKXWE+++/n/vvv7/Vt3s8XW0MABKbOxCllFJtyzFrECJSjNUGIfbrAeC+Fo5LKaWUjzXmFFNEawSilFKqbak3QYjIqIYWNMasbf5wlFJKtRUN1SCebWCaASY3cyxKKdUos2bNIjw8nHvuuYc5c+Zw1lln0b179yat48UXXyQ0NJTrrruuUfNnZmYyZMgQBg0ahDGGsLAwXnvtNQYNGgTAF198wf/+7/9SVlZGUFAQkydP5tlnn2XWrFnMnj37iL6TFi1aRHR0dJPi9YV6E4Qx5ozWDEQppY7HnDlzSElJ8ZogXC5XvXcd33LLLU3eVr9+/Vi/fj0A//rXv3jyySd5/fXX2bRpE7fddhv//e9/GTx4MC6X64geWu+66y7uueeeJm/P1xp1FZOIpIjIFSJyXc3QyOWmicg2EdkpIl6v0bLXu1lEMkTkbY9yl4ist4dPG7c7SqmOylt33x988AHp6elcffXVjBgxgvLycpKSkrjvvvsYNWoU77//fr3deHs+dGjSpEncd999jBs3joEDBzbqZrSioiJiYmIA+POf/8yDDz5Yewe2v79/bQd/7VljrmL6IzAJGAp8DpyDdR/EG8dYzh94AZgKZAGrReRTY8xmj3kGAH8ATjbG5IuI5+Wz5caYEU3aG6VUi/vTqj+x9fDWZl3n4NjB3Deu/osj6+vu+7LLLuP555/nmWeeYcyYn28GjouLY+1aq5k0Ly/PazfedVVXV7Nq1So+//xzHnnkEebPn3/UPD/++CMjRoyguLiYsrIyVq5cCVg9wd599931xv+Xv/yFN998E4CYmBgWLlzYiE/F9xpzJ/VlQBqwzhhzo4h0Ad5sxHLjgJ3GmF0AIjIXuBDY7DHPzcALxph8AGNMblOCV0p1Dp7dfQNHdPftjWcfRps2beKhhx6ioKCAkpISzj77bK/LXHLJJQCMHj2azMxMr/N4nmJ69913mTlzJl9++eUx42+vp5gakyAqjDFuEakWkUggF+jViOV6AJ7PkcgCxteZZyCAiCwF/IFZxpiaTztYRNKBauApY8wnjdimUqqFNXSk31Z4dpLX2G68g4KCAOv0UGMeCzp9+vTajvOGDRvGmjVrSEtLO/Hg25B62yBE5AUROQVYJSLRwGxgDbAWWN5M2w/AujN7EjADmG1vC6CP3YHUL4C/ikg/LzHOFJF0EUlvjcfvKaV8o6HuviMiIiguLq532fq68T5RS5YsoV8/62fp3nvv5cknn2T79u2A9SyIF198sdm25SsN1SC2A08D3YFS4B2s9oRIY8yGRqw7myNrGj3tMk9ZwEpjjBPYLSLbsRLGamNMNoAxZpeILAJGAj96LmyMeQl4CazeXBsRk1KqHfLs7jsxMfGI7rtvuOEGbrnlFkJCQli+/Ohj15puvBMSEhg/fnyDyeRYatogjDEEBgby8ssvAzB8+HD++te/MmPGDMrKyhARzj///NrlPNsgAD755BOSkpKOO47W0pjuvvsAV9lDCFaieNsYs+MYywVgJZkpWIlhNfALY0yGxzzTgBnGmOtFJB5Yh/VAIjdQZoyptMuXAxd6NnDXpd19K9VytLvvjqGp3X035oFBe4wxfzLGjMQ6DXQRcMxLGIwx1cBtwFdYjyh9zxiTISKPikhNC9NXQJ6IbAYWAvfaDyMaAqSLyA92+VMNJQellFLNrzGXuQZgXdp6FVZtYBEwqzErN8Z8jnVprGfZwx7vDfA7e/CcZxmQ2phtKKWUahkN9cU0FavGcC6wCpgLzDTGlLZSbEoppXyooRrEH4C3gbtr7lNQSinVeTTUF1On6YzPVVKCf3i4r8NQSqk25XieKNehVO3bx65zziX/vfd8HYpSSrUpnT5BBHTpQtDQIRz44ywKPvrY1+EopRrBs6O9prrhhhv44IMPvJYnJyczYsQIBg8ezCOPPFI7raSkhP/5n/+hX79+jB49mkmTJtX2w+Tv78+IESNqh6eeeur4dqoNakxXGx2aX2AgPf/2N7Ju/TU/PfggEuBP1DH6eVFKdUxPP/00l112GRUVFQwdOpTrrruO5ORkbrrpJpKTk9mxYwd+fn7s3r2bzZutK+9DQkJq+2fqaDp9DQLALyiIni88T+i4cey//w8UffGFr0NSStXhrbvvrVu3Mm7cuNp5MjMzSU21rpB/9NFHGTt2LCkpKcycOZNj3RTsqaKiArD6dPrxxx9ZuXIljz/+OH5+1k9mcnIy5513XnPtWpvV6WsQNfxCQuj1z3+wd+ZMsu+5F/z9iTzrLF+HpVSbc+DJJ6nc0rzdfQcNGUzXBx6od3p93X0PHjyYqqoqdu/eTXJyMu+++25tT6633XYbDz9s3XZ17bXXMm/ePC644IIG47j33nt5/PHH2blzJ3fccQeJiYmsWLGCESNG1PvgofLyckaMGFE7/oc//OGI3mTbM61BePALDaXXi/8iJDWV7N/dTfGC9tFnu1IdnWd335GRkUd0933FFVfw7rvvAhyRIBYuXMj48eNJTU1lwYIFZGRkeF23p6effpr169dz4MABvv32W5YtW3bMZWpOMdUMHSU5gNYgjuIfHkav2S+x91c3kf3b39L96aeJnOa9/3ilOqOGjvR94corr+Tyyy/nkksuQUQYMGAAFRUV/PrXvyY9PZ1evXoxa9as2tNGjREeHs6kSZNYsmQJl156KT/88EODjy/tqLQG4YV/RAS9Z79E0JAhZN95J9n33Et1vt4rqJSvNNTdd79+/fD39+exxx6rPXqvSQbx8fGUlJR4vWqpIdXV1axcuZJ+/frRr18/xowZwx//+MfadozMzEz++9//NtPetV2aIIBq99EPB/GPiiLpzX8Tf/ttFH31FbvOv4CiL7/yQXRKKc/uvs8555wjuvsGqxbx5ptvcsUVVwAQHR3NzTffTEpKCmefffZR89fn3nvvZcSIEQwfPpzU1NTap8y9/PLL5OTk0L9/f1JSUrjhhhtITLSekFzTBlEz3H///c245751zO6+24vj7e47tyyXmV/P5PaRtzOlzxSv81Rs285PDzxARUYGEWedRdeH/5eA+PgTDVmpdkO7++4Ymr27744uNCCU8MBw7vnuHubvOfoh5QDBgwaS9O5cEn73O0oWLWLXeedT8NHHGJerlaNVSqnW0+kTRHhgOC+e+SIp8Snc+929fLPnG6/zSUAA8TNvJvnjjwhMTuanBx5g90UXUfTNN026vloppdqLTp8gwE4SU4+dJACC+vWjz9tv0eMv/4epdpF9+x1kXn4FJUuWaqJQSnUomiBsYY4wXpz6IsMThnPvd/fyVWb9DdLi50fkOefQ97NP6fbEE7gOH2bfTTex99rrKFuzphWjVqr16AFQ+3Y8fz9NEB7CHGH888x/kpaQxn2L7+PLzC8bnF8CAoi+9BL6fvkFXR56iMrMTPZcfQ2Zv7ia4vnzMW53K0WuVMsKDg4mLy9Pk0Q7ZYwhLy+P4ODgJi3X6a9i8qbMWcat82/lh4M/8PBJD3Nx/4sRkWMu5y4ro+DDjzg8Zw7O7GwCk5KIvfFGoi6cjl8T/zBKtSVOp5OsrKwm3Wym2pbg4GB69uyJw+E4oryhq5g0QdSjzFnGHQvvYOVPK5ncazIPn/QwcSFxjVrWVFdT/PXX5L3yKhUZGfjHxhJzzdXEXHGFXh6rlGpTNEEcJ5fbxZtb3uS5tc8RERjBwxMervdeCW+MMZStWk3eq69Q+t1icDiIPPtsYn4xg5CRIxtVK1FKqZakCeIE7czfyQNLHmDL4S1M7zed+8fdT0RgRJPWUblrF/lz51L48Se4i4sJGjSImBkziLrgfPzCwlokbqWUOhZNEM3A6Xby0oaXmL1hNgmhCTxy0iNM7DGxyetxl5VROG8e+W+/Q+XWrfiFhRF5wflEX3oZwSnDtFahlGpVmiCa0caDG3lgyQNkFmUypfcU7h5zN70iejV5PcYYytevJ/+ddyj+6mtMZSVBgwYRfeklRF5wAQExMS0QvVJKHUkTRDOrdFXyRsYbzN44G5fbxfXDruem1JsIdYQe1/pcRUUUff45BR98SMWmTYjDQfiUKURfcjFhEyciAdoru1KqZWiCaCE5pTk8t/Y5Ptv1GQkhCdw1+i7O63sefnL8t5dUbNtGwYcfUvSfT3EVFuIfF0fkeecSNf1CgocN1VNQSqlmpQmiha3PXc+fVv2JTXmbGB4/nDtH38nYro3rXrg+7qoqShcvpvDTzyhZuBDjdBLYty9R0y8g8vwLCOzZo5miV0p1ZpogWoHbuPn0x0/5+7q/k1uWy8ndT+aOUXcwNG7oCa/bVVhI0VdfUfTpZ5TZ+xicNpzIc84hcto0HF27nvA2lFKdkyaIVlRRXcHcrXN5edPLFFYWclafs7ht5G0kRyU3y/qrsrIp+uJzir74gsrNWwAIGTWKyHPOIeLss3DYDzFRSqnG0AThA8VVxbye8TpvbH6DKlcVF/W/iJnDZ9I9vHuzbaNy926Kv/ySos+/oHLHDhAhZNQoIqaeScSZU/U0lFLqmDRB+NCh8kO8vPFl3tv2HgbDhf0u5ObhN9MjvHl/vCt37qToy68onj+fyq1bAQgeOpSIs6YSceaZBPbrpw3cSqmjaIJoAw6UHuDljS/z0Y6PMMYwvf90bkq96bjuoTiWqr17Kf5mPsXffEP5+vUAOPr0JmLSGYSfcQaho0chdTrsUkp1Tpog2pADpQd4ddOrfLj9Q1zGxQX9LuCm1JvoE9mnRbbnzMmh+NtvKVm4iLIVKzBOJ36RkYSfeirhZ5xB+Kmn4B8V1SLbVkq1fZog2qDcslxe3fQq7297n2pTzdQ+U/llyi+b5aqn+rhKSildtpSShYso+e47XIcPg58fIWlphJ92KmGnnkbw0CGInz4mRKnOQhNEG3ao/BBvbn6Td7e9S4mzhIndJ/KrlF8xtuvYFm0zMC4X5Rs2ULJ4MaWLv6ciIwMA/7g4wk85mbBTTiVs4kkExDWui3OlVPukCaIdKK4q5r1t7/Hvzf8mryKP1PhUbky5kcm9JuPv59/i26/Oy6N0yRJKFn9P6ZIluAoLAQgaPJiwiRMJmziR0DGj9cFHSnUwmiDakUpXJf/Z+R/mZMxhX/E+eoT34Joh13DxgIsJc7ROt+DG5aJi82ZKly6jdNkyytatA6cTCQwkZNQowsaPI3T8eEJSUpDAwFaJSSnVMjRBtEMut4tF+xbxxuY3WJu7lnBHOJcOuJSrh1xNt/BurRqLu6yMsvR0Spctp3T5ciq3bQNAQkIIHTWK0PHjCRs/juChQ/XqKKXaGU0Q7dzGgxv59+Z/8/WerwE4s8+ZzBg8g1GJo3xyb0N1fj5lq1dTtnIVZatWUrljJ2AljJARaYSOHkPomDGEpA3HLySk1eNTSjWezxKEiEwDngP8gZeNMU95mecKYBZggB+MMb+wy68HHrJne9wY83pD2+rICaLGgdIDvL3lbT7Y8QHFVcUMjBnIjMEzODf53OPuarw5VOflWQkjfQ1l6elWDcMYcDgIGTaMkFGjCBk5gtARIwhISPBZnEqpo/kkQYiIP7AdmApkAauBGcaYzR7zDADeAyYbY/JFJNEYkysisUA6MAYrcawBRhtj8uvbXmdIEDXKq8v5fNfnvLP1HbblbyMiMIKL+1/MVYOuoldk899411SuoiLK1q6lfM0aytLXULFpE8bpBMDRq5eVLEaOJCQtjaABA/S0lFI+5KsEcRIwyxhztj3+BwBjzP/zmOfPwHZjzMt1lp0BTDLG/I89/i9gkTHmnfq215kSRA1jDOty1/HO1neYv2c+1aaak7qdxGUDL+OM3mfg8GsbP7zuqioqMjIoX7ee8nXrKFu/DtfBQwBIUBDBw4YRkppKSNpwgocPx9Gjh3YLolQraShBtOSjynoA+zzGs4DxdeYZCCAiS7FOQ80yxnxZz7JHdV4kIjOBmQC9e/dutsDbCxFhVJdRjOoyityyXD7c8SEf7fiIu7+7m7jgOC7qfxGXDrjU57UKv8BAQkeOJHTkSOBGjDE4s7Op2LCB8h82UL5hA/lz53L4desson9MDMHDhhGcMsxKHikpBHTtqklDqVbWkjWIy4Bpxpib7PFrgfHGmNs85pkHOIErgJ7AYiAVuAkINsY8bs/3v0C5MeaZ+rbXGWsQ3rjcLpbuX8oH2z9gcdZiXMbFhG4TuGTAJUzuPZkg/yBfh+iVcTqp2L7dShqbNlGRsdnqodblAqwb+IKHDSV48BCChw4hePBgHL17613fSp0gX9UgsgHPQ9eedpmnLGClMcYJ7BaR7cAAe75JdZZd1GKRdiD+fv6c1vM0Tut5GjmlOXy882M+2vERv1/8eyICIzg3+Vwu6n8Rw+KGtakjcqlp0B42jBi7zF1RQeW2bZRnZFCxKYOKjAzyli2H6moA/EJDCRo8mODBgwkaNIjgQQMJGjAAv7DWuV9EqY6uJWsQAViN1FOwfvBXA78wxmR4zDMNq+H6ehGJB9YBI/i5YXqUPetarEbqw/VtT2sQ9XMbN6sOrOLjHR/z7d5vqXRV0j+6Pxf1v4jz+p5HfEi8r0NsNHdVFVU7d1KxZQsVW7ZSsWULlVu34i4trZ3H0asXQYMGEjxwEEED+hPUvz+BSUnaGK6UF768zPVc4K9Y7QuvGmOeEJFHgXRjzKdiHcI+C0wDXMATxpi59rK/BB6wV/WEMea1hralCaJxiqqK+HL3l/xn53/YcGgD/uLPhO4TOL/v+UzuNdmnl8seL+N249y/n8rt26ncto2Kbdup3L6dqsxMcLutmRwOgpL6ENjfShhB/foT2DeZwKQk/PRucNWJ6Y1yyqtdBbuYt2se/931X/aX7ickIIQpvadwft/zGd9tPAF+LXkGsuW5Kyqo2rWLyp07qdyx03rduRNnVpZ1nwaAnx+OXj0J6tuPwL7JBPXtS2BSEoFJSfjHxrap03BKtQRNEKpBbuNmXe465u2ax1eZX1FcVUxscCxT+0zlnORzGJk4Ej/pOI3B7rIyqjIzqfxxF1W7d1mvu3ZRlZlZe78GgF9ERG2yCEzqQ2DvPgT26U1g7974R0f7bgeUakaaIFSjVbmqWJy1mC92f8HirMVUuCpIDE3k7KSzmZY0jdT41A57VG2qq3Hu309VZuYRQ2VmJtX7fzpiXr+oKAJ79yawVy8cvXsR2LMnjp69COzV07ok17/le+BVqjloglDHpcxZxqJ9i/gy80uWZC/B6XbSPaw7Z/Y5k6l9pjI8YXiHqlk0xF1RgXPfPqr27aNqz16q9u7BuXcfVXv34ty/v/ZyXAACAnB0746jR3ccPXrg6N6dQPvV0aMHAYmJSED7Pn2nOg5NEOqEFVUVsXDvQr7e8zXL9y/H6XaSEJLA5N6TmdpnKqO7jG73bRbHy1RX4zyQgzPLSiDOfVnW++xsnPv31941Xsvfn4DERBzdutlDVwLs9wFduuDo2hX/mBi9x0O1Ck0QqlkVVxWzOGsx8/fMZ0n2EipcFUQHRXNaz9OY3GsyJ3U/qV1eDdVS3JWVOPfvx5m9H2d2Ns4DP1G9/yecP9nDgQPg0fYB1n0hAV26ENC1C47ELtb7xEQCEhNw1L5P1Ac4qROmCUK1mDJnGUv3L+Xbvd+yOGsxxVXFBPoFMqH7BM7odQan9zydhFDtwbUhxu3GlZdnJYucHKoP5FCdcwBnTi7VBw5YZTk5mMrKo5b1i4ggID6egISEn18T4vGPiycgPo6AuDjrfWyM3geivNIEoVqF0+1kXc46Fu5byMJ9C8kusW6cHxY3rPbu7qFxQztNu0VzMsbgLi6mOicHZ24u1bkHqc7JofrQIWs4eJDqQwepPngIU1bmdR3+0dH4x8UREBuLf2wsAXGx+MfE4h8bYyWS6Bj8Y2Lwj4kmIDpanxbYSWiCUK3OGMOOgh0s2reIxVmL2XBwAwZDbHAsp/Q4hVN7nsrE7hOJDIz0dagdjqukFFfeIarz8qg+dAhXXh7Vh/KozjuEK+8w1fmHcR3Ox5WXV/vscW/8wsOthBEdbQ1RUXXeR1mvkZH41UyLiNAG+HZGE4TyufyKfJbuX8r3Wd+zdP9SCisL8RM/UuNTObn7yUzsMZGUuBT8/fTy0NZkqqtx5edTfTgfV0EBrvzD1nh+Pq78Alz5dnlhofVaUIC7uLjBdfqFheEXGYl/ZCT+ERHW+9rXcPzCI/CLCLfKIiKs1/Bw/MLC8Q8PQ0JDO+yl1G2RJgjVplS7q9l4aCNLs5eybP8yNh3ahMEQERjBhG4TmNh9IuO7jadXhO8ffqSOZqqrcRUV4SooxF1UaCUPe9xVWIi7uAhXUTGu4iLchUW4iotxF9mvJSU/38VeHz8/K2GEh+EfFoZfaJiVdOoOoaHWEBZa+15CQqz5Q0PwC7GH0FA9XdYATRCqTSuoKGDFgRUsy17G0v1LyS3LBaBHeA/GdxvPhG4TGNd1HHEhcT6OVJ0o43bjLiuzE0YJ7pJiK3GUluIuLsFdWoKrpAR3SSnumvKaoawUV2mpNa2srLZX30YJCMAvOBi/kBAriYSE4BccjISG4BcUjIQEH/kaHGRNDwrGLzjo59fgYCQwCL+gQCQoyBoCA/HzeC9BQYjD0W5qQZogVLthjGF30W5W/rSSFftXsDpnNcVV1imN/tH9Gdt1LGO7jmV0l9HEBsf6OFrlK8YYjNOJu7QUU1ZmJZ3SUtzl5dZQVo67rBTjOV5RjimvsMY93puKCtwVFbWv7ooKTHn5sWs6xyCBgUcPDof93oGfw3rF4bDKHQ6r40iHAwkIQByB1mtAgD09AAICkICa6dY0AgIIiIsn/NRTji9OTRCqvXK5XWw5vIUVP61g1U+rWH9wPeXV5YCVMMZ0GcOYrmMY3WV0u+q2XLVtNQnIVFZaiaPmtaISU1WJqay0yiqrMFWVVlKpqsJU2ctUVWGcVdY8NeVVVdY6q6p+Hqqrfy6vGWrKq6ut8erqo+6TqSskLY2kd+ce175qglAdhtPlJCMvg/ScdFYfWM263HW1CaN3RG/rEayJoxiZOJI+kX3aTTVfqYYYY6BO0jDOaqi23ktAAI4eRz2VuVE0QagOy+l2sjlvM+ty1rEmdw3rctdRWGlduhkbHMuIhBGMSLSGoXFD2+wjV5XyFU0QqtNwGze7C3ezNncta3PW8sPBH9hXvA+AAL8AhsYOZXjCcNIS0khNSKV7WHetZahOTROE6tTyyvP44eAP/HDwB9bnricjL4NKl9VtRWxwLKnxqaTEpzA8fjjD4ocRFRTl44iVaj2aIJTy4HQ72ZG/g40HN7LxkDXsKtxVO71neE+GxQ9jaNxQhsUNY0jcEL3jW3VYmiCUOobiqmIy8jLYdGgTm/M2szlvc21fUmA1gA+OHcyQuCEMjh3M4NjBetWU6hAaShDaaYpSUHsX94RuE2rL8ivy2ZK3hYy8DDbnbSYjL4Ov93xdOz0hJIEhcUMYFDOIgTEDGRg7kD4RfbS7ENVhaIJQqh4xwTFM7DGRiT0m1pYVVRWx7fA2NudtZuvhrWw9vJWl2UtxGeuJckH+QfSP7s+g2EEMiB5A/5j+DIgeoHeBq3ZJTzEpdYIqXZXsKtjFtvxtbM/fzvbD29mWv42CyoLaeWKDY+kf3Z8BMQPoF92PflH96BfdTxvElc/pKSalWlCQfxBD4oYwJG5IbZkxhryKPHbk72Bnwc7a1492fFR7Yx9AXHAc/aL70TeqL/2i+5EclUxyVDIJIQl6+a3yOU0QSrUAESE+JJ74kHhO6n5SbbnbuPmp9Cd+LPiRXQW7+LHQev1s12eUOktr5wtzhJEcaSWLpKgkkiKT6BPZh96RvQkJCPHFLqlOSE8xKdUGGGPIKcthd+FuMosy2V24u3bIKcs5Yt6uYV3pE9mHpMgkekX0shJHRG96RPTQO8VVk+kpJqXaOBGha1hXuoZ1PaLGAdZzv/cW7yWzKJM9hXvYU2QNn+/+vLanWwDBWkfviN70jOhZO/SK6EXP8J7a3qGaTBOEUm1cqCO09t6LugorC9lTtIe9xXvZV7SPvcV72Vu0l4X7FnK44vAR80YERtAz3Eoa3cO60yOiBz3CraF7eHc9daWOoglCqXYsKiiK4QnDGZ4w/Khppc5SsoqzrKEki33F+8gqyWJH/g6+2/cdVe6qI+aPDY6lW1g3uod3P+K1ZogKitKG805GE4RSHVSYI4xBsYMYFDvoqGlu4yavPI/skmyyS7LZX7Kf/aX72V+ynx35O/g+63sqXBVHLBPsH1x7Gqxm6BLaxRrCrNfIwEhNIh2IJgilOiE/8SMhNIGE0ARGJI44aroxhsMVh/mp9CcOlB7gQOmBn9+XHWBZ9jIOlh/EcORFLiEBISSGJv48hCQeMZ4QmkB8SLw2prcTmiCUUkcREeJC4ogLiSMlPsXrPE63k7zyPA6UHiCnLKf2Nbcsl9yyXNbnrie3LBen++inoUUFRZEQYiWLxNBE4kLiiA+2LgtOCE2wxkPiiXBEaI3EhzRBKKWOi8PPUXuqqT7GGAoqC2qTxqHyQxwsP/jz+7KDrD6wmkPlh7wmEoefw0pUwXFHvcYGxxIbEmu9BscSHRRNgJ/+pDUn/TSVUi1GRIgJjiEmOMZrW0gNYwxFVUUcKj90xJBXnkdeRR555XnkluWyJW8LhysO1/Z9dcS2EKKCoqztBcUQGxxbu+3Y4FhrWlAM0cHR1mtQNCEBIVpDaYAmCKWUz4lYP+5RQVH0i+7X4Lxu46a4qpi8ijwOlx/mcIU15FXkkV+Rbw2V+WQWZbI2dy0FlQW4jdvrugL9AokOiiYqOIrooGjrfdDP7yMDI2vjigqMIjrYKgv0D2yJj6HN0QShlGpX/MSv9ke7b1TfY87vNm6KKosoqCygoLKA/Ip867Uyn4KKgtrywspCfiz4sfa9t1pKjWD/YCIDI4kMijzy1R4iAiOIDIokwhFBRKA1RAZGEh4YTpgjDD/xa86PpMVoglBKdWh+4kd0cDTRwdGNXsYYQ6mzlMKqQgorjxyKqopqX2uG/SX72Va1jaKqoiP61PJGEMIDw4lwRBAeGE64I5yIwCPfhznCiHBEEBZovzrCapNLuMN6bY1ajCYIpZSqQ8T6EQ8PDKdHeI8mLVvtrqbUWUpRZRFFziKKKosocZZQXFV89OAspqSqhJyyHHYW7KTEWUJJVUmDtZcaDj8H4Y5wQh2hpMan8vTpTx/v7tZLE4RSSjWjAL+A2lNgx8MYQ4WrglJnKcVVxZQ6S2sTR837Umdp7VDiLKFbWLdm3guLJgillGpDRISQgBBCAkJ8/tzzFm0pEZFpIrJNRHaKyP1ept8gIgdFZL093OQxzeVR/mlLxqmUUupoLVaDEBF/4AVgKpAFrBaRT40xm+vM+q4x5jYvqyg3xoxoqfiUUko1rCVrEOOAncaYXcaYKmAucGELbk8ppVQzaskE0QPY5zGeZZfVdamIbBCRD0Skl0d5sIiki8gKEbnI2wZEZKY9T/rBgwebL3KllFIt2wbRCJ8BScaY4cA3wOse0/rYj8H7BfBXETnq9kpjzEvGmDHGmDEJCQmtE7FSSnUSLZkgsgHPGkFPu6yWMSbPGFNpj74MjPaYlm2/7gIWASNbMFallFJ1tGSCWA0MEJFkEQkErgKOuBpJRDwv3p0ObLHLY0QkyH4fD5wM1G3cVkop1YJa7ComY0y1iNwGfAX4A68aYzJE5FEg3RjzKXCHiEwHqoHDwA324kOAf4mIGyuJPeXl6iellFItSIwxx56rHRCRg8CeE1hFPHComcJpT3S/Oxfd786lMfvdxxjjtRG3wySIEyUi6XajeKei+9256H53Lie6376+ikkppVQbpQlCKaWUV5ogfvaSrwPwEd3vzkX3u3M5of3WNgillFJeaQ1CKaWUV5oglFJKedXpE8SxnlnRkYjIqyKSKyKbPMpiReQbEdlhv8b4MsbmJiK9RGShiGwWkQwR+a1d3tH3O1hEVonID/Z+P2KXJ4vISvv7/q7dy0GHIyL+IrJORObZ451lvzNFZKP9HJ10u+y4v+udOkF4PLPiHGAoMENEhvo2qhY1B5hWp+x+4FtjzADgW3u8I6kG7jbGDAUmAL+x/8Ydfb8rgcnGmDRgBDBNRCYAfwL+YozpD+QDv/JdiC3qt9hd99g6y34DnGGMGeFx/8Nxf9c7dYKgkz2zwhizGKtLE08X8nMvuq8DF7VmTC3NGPOTMWat/b4Y60ejBx1/v40xpsQeddiDASYDH9jlHW6/AUSkJ3AeVgegiIjQCfa7Acf9Xe/sCaKxz6zoyLoYY36y3x8AuvgymJYkIklYvQKvpBPst32aZT2Qi9Wd/o9AgTGm2p6lo37f/wr8HnDb43F0jv0G6yDgaxFZIyIz7bLj/q63WGd9qv0xxhgR6ZDXPYtIOPAhcKcxpsg6qLR01P02xriAESISDXwMDPZtRC1PRM4Hco0xa0Rkko/D8YVTjDHZIpIIfCMiWz0nNvW73tlrEMd8ZkUnkFPT7br9muvjeJqdiDiwksNbxpiP7OIOv981jDEFwELgJCBaRGoODDvi9/1kYLqIZGKdMp4MPEfH32/giOfo5GIdFIzjBL7rnT1BHPOZFZ3Ap8D19vvrgf/4MJZmZ59/fgXYYoz5P49JHX2/E+yaAyISAkzFan9ZCFxmz9bh9tsY8wdjTE9jTBLW//MCY8zVdPD9BhCRMBGJqHkPnAVs4gS+653+TmoRORfrnGXNMyue8G1ELUdE3gEmYXUBnAP8EfgEeA/ojdVd+hXGmLoN2e2WiJwCfA9s5Odz0g9gtUN05P0ejtUg6Y91IPieMeZREemLdWQdC6wDrvF4qmOHYp9iuscYc35n2G97Hz+2RwOAt40xT4hIHMf5Xe/0CUIppZR3nf0Uk1JKqXpoglBKKeWVJgillFJeaYJQSinllSYIpZRSXmmCUKoJRMRl95RZMzRbJ38ikuTZ065SvqZdbSjVNOXGmBG+DkKp1qA1CKWagd0P/5/tvvhXiUh/uzxJRBaIyAYR+VZEetvlXUTkY/t5DT+IyER7Vf4iMtt+hsPX9l3QSvmEJgilmiakzimmKz2mFRpjUoHnse7OB/g78LoxZjjwFvA3u/xvwHf28xpGARl2+QDgBWPMMKAAuLRF90apBuid1Eo1gYiUGGPCvZRnYj2gZ5fdOeABY0yciBwCuhljnHb5T8aYeBE5CPT07O7B7o78G/vBLojIfYDDGPN4K+yaUkfRGoRSzcfU874pPPsHcqHthMqHNEEo1Xyu9Hhdbr9fhtWrKMDVWB0HgvXox1uh9sE+Ua0VpFKNpUcnSjVNiP2UthpfGmNqLnWNEZENWLWAGXbZ7cBrInIvcBC40S7/LfCSiPwKq6ZwK/ATSrUh2gahVDOw2yDGGGMO+ToWpZqLnmJSSinlldYglFJKeaU1CKWUUl5pglBKKeWVJgillFJeaYJQSinllSYIpZRSXv1/PcT6Ouvgwm8AAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"model.plot(path=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Strictly speaking, the Perceptron seems to have converged in the right direction."
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"By reducing the learning rate, all other things being equal, we obtained greater accuracy, lower cost and smoother curves on the plot.\n",
"\n",
"You may have observed something possibly counter-intuitive: \n",
"\n",
"* The cost, which describes the mean difference between **output probabilities** and labels, is lower for the validation set compared to training set.\n",
"* The accuracy, which describes the mean difference between **output decisions** and labels, is higher for the validation set compared to training set.\n",
"\n",
"While the *cost* says the error is higher when evaluating on the validation set, the *accuracy* says the opposite.\n",
"\n",
"That’s because accuracy compares **decisions** and labels, whereas the cost from the loss function compares **probabilities** and labels.\n",
"\n",
"For code, maths and pictures behind the *Flatten* and *Dense* layers, follow these links:\n",
"\n",
"* [Flatten - Adapter](https://epynn.net/Flatten.html)\n",
"* [Fully Connected (Dense)](https://epynn.net/Dense.html)\n",
"\n",
"Let’s take a break and understand the difference in addition to making clear some semantics."
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"### Difference between accuracy and cost"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"The question is: Can we expect identical costs for the training and validation set **if** the accuracy is identical for the training and validation set?\n",
"\n",
"Let’s compare what outputs from the *dense* layer (A) to the set of sample label (Y)."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[0.24990897 0.75009103]\n",
" [0.15080179 0.84919821]\n",
" [0.28627148 0.71372852]\n",
" ...\n",
" [0.09204908 0.90795092]\n",
" [0.15813641 0.84186359]\n",
" [0.41724774 0.58275226]]\n",
"(320, 2)\n",
"[[0. 1.]\n",
" [0. 1.]\n",
" [1. 0.]\n",
" ...\n",
" [1. 0.]\n",
" [0. 1.]\n",
" [0. 1.]]\n",
"(320, 2)\n"
]
}
],
"source": [
"# This is probability distributions for each sample. \n",
"print(model.embedding.dtrain.A)\n",
"print(model.embedding.dtrain.A.shape)\n",
"\n",
"# These are the labels we target\n",
"print(model.embedding.dtrain.Y) \n",
"print(model.embedding.dtrain.Y.shape) "
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"We have probabilities (A) versus binary values (Y).\n",
"\n",
"To compute the accuracy, one needs to convert probabilities to decisions, as well as to retrieve single-digit labels."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1 1 1 ... 1 1 1]\n",
"[1 1 0 ... 0 1 1]\n"
]
}
],
"source": [
"print(np.argmax(model.embedding.dtrain.A, axis=1))\n",
"\n",
"# Equivalent to calling model.embedding.dtrain.y directly\n",
"print(np.argmax(model.embedding.dtrain.Y, axis=1))"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Then, accuracy is computed such as:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ True True False ... False True True]\n",
"0.740625\n"
]
}
],
"source": [
"print ((np.argmax(model.embedding.dtrain.A, axis=1) == model.embedding.dtrain.y))\n",
"print ((np.argmax(model.embedding.dtrain.A, axis=1) == model.embedding.dtrain.y).mean())"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"The cost is computed from **probabilities, not from decisions**. This apart from the fact that accuracy and cost are simply two different functions.\n",
"\n",
"To compute a cost, we first need to compute the loss, which provides information for each single probability in the array (A)."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(320,)\n"
]
}
],
"source": [
"# This is the cost. It deals with \"true\" labels against probabilities\n",
"\n",
"loss = model.training_loss(model.embedding.dtrain.Y, model.embedding.dtrain.A)\n",
"\n",
"print(loss.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"The cost is a form of average of the loss. Whereas the loss is an array from element-wise comparison between probabilities and labels, the cost is a scalar which is an average per sample, itself an average of the element-wise loss for this sample."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(320,)\n",
"()\n",
"0.5554707080208504\n"
]
}
],
"source": [
"print(loss.shape) # Averaged for each sample\n",
"print(loss.mean().shape) # Average of above - scalar\n",
"\n",
"cost = loss.mean()\n",
"\n",
"print(cost)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Note that what is fed back in the network during the backward propagation phase is not the loss. It is the **derivative** of the loss."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.2875607 0.16346266 1.2508147 ... 2.38543339 0.17213728 0.53999313]\n",
"[[ 0.66658576 -0.66658576]\n",
" [ 0.58879069 -0.58879069]\n",
" [-1.74659385 1.74659385]\n",
" ...\n",
" [-5.43188494 5.43188494]\n",
" [ 0.59392045 -0.59392045]\n",
" [ 0.85799753 -0.85799753]]\n"
]
}
],
"source": [
"dloss = model.training_loss(model.embedding.dtrain.Y, model.embedding.dtrain.A, deriv=True)\n",
"\n",
"print(loss)\n",
"print(dloss) # dloss is referred to as dA"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"The loss function and derivatives natively provided with EpyNN can be found in `EpyNN/epynn/commons/loss.py`.\n",
"\n",
"The metrics natively provided with EpyNN can be found in `EpyNN/epynn/commons/metrics.py`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Recurrent Architectures"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Herein, we are going to chain simple schemes based on recurrent architectures.\n",
"\n",
"There are three most commonly cited recurrent layers:\n",
"\n",
"* **Recurrent Neural Network (RNN)**: This is the most simple recurrent layer. It is composed of one to many recurrent units. Each cell performs a single activation which outputs the *hidden cell state* or simply *hidden state*.\n",
"* **Long Short-Term Memory (LSTM)**: By contrast with the RNN cell, the LSTM cell requires four activations which correspond to three different gates: forget, input (two activations), and output. To compute the hidden cell state, it then requires a fifth activation. Note that in addition to the hidden cell state, there is another so-called cell *memory* state.\n",
"* **Gated Recurrent Unit (GRU)**: Compared to the LSTM cell, the GRU cell has only two gates: reset and update. Practically talking, GRU trains faster than LSTM and is reported to perform better on small datasets or shorter sequences. Both GRU and LSTM, however, are state-of-the-art architectures to deal with sequential data.\n",
"\n",
"See [here](epynnlive/dummy_time/train.html#Recurrent-Neural-Network-(RNN)) for more detailed practical descriptions or simply via the pages linked on top of this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Embedding"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, we use the same setup as for Feed-Forward networks."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"embedding = Embedding(X_data=X_features,\n",
" Y_data=Y_label,\n",
" X_encode=True,\n",
" Y_encode=True,\n",
" relative_size=(2, 1, 0))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now chain the simplest schemes to train binary classifiers based on recurrent layers."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### RNN-Dense"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The number of RNN units in the RNN layer is set to 1."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"name = 'RNN-1_Dense-2-softmax'\n",
"\n",
"se_hPars['learning_rate'] = 0.001\n",
"\n",
"rnn = RNN(1)\n",
"\n",
"dense = Dense(2, softmax)\n",
"\n",
"layers = [embedding, rnn, dense]\n",
"\n",
"model = EpyNN(layers=layers, name=name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Initialize the model."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m--- EpyNN Check OK! --- \u001b[0m\r"
]
}
],
"source": [
"model.initialize(loss='BCE', seed=1, se_hPars=se_hPars.copy(), end='\\r')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Train for 50 epochs."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m\u001b[37mEpoch 9 - Batch 0/0 - Accuracy: 0.734 Cost: 0.58678 - TIME: 0.94s RATE: 1.06e+01e/s TTC: 0s \u001b[0m\n",
"\n",
"+-------+----------+----------+----------+-------+--------+-------+----------------------------------+\n",
"| \u001b[1m\u001b[37mepoch\u001b[0m | \u001b[1m\u001b[37mlrate\u001b[0m | \u001b[1m\u001b[37mlrate\u001b[0m | \u001b[1m\u001b[32maccuracy\u001b[0m | | \u001b[1m\u001b[31mBCE\u001b[0m | | \u001b[37mExperiment\u001b[0m |\n",
"| | \u001b[37mRNN\u001b[0m | \u001b[37mDense\u001b[0m | \u001b[1m\u001b[32mdtrain\u001b[0m | \u001b[1m\u001b[32mdval\u001b[0m | \u001b[1m\u001b[31mdtrain\u001b[0m | \u001b[1m\u001b[31mdval\u001b[0m | |\n",
"+-------+----------+----------+----------+-------+--------+-------+----------------------------------+\n",
"| \u001b[1m\u001b[37m0\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.544\u001b[0m | \u001b[1m\u001b[32m0.581\u001b[0m | \u001b[1m\u001b[31m0.676\u001b[0m | \u001b[1m\u001b[31m0.659\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m1\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.648\u001b[0m | \u001b[1m\u001b[31m0.630\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m2\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.629\u001b[0m | \u001b[1m\u001b[31m0.610\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m3\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.615\u001b[0m | \u001b[1m\u001b[31m0.595\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m4\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.605\u001b[0m | \u001b[1m\u001b[31m0.585\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m5\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.598\u001b[0m | \u001b[1m\u001b[31m0.577\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m6\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.593\u001b[0m | \u001b[1m\u001b[31m0.571\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m7\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.589\u001b[0m | \u001b[1m\u001b[31m0.566\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m8\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.587\u001b[0m | \u001b[1m\u001b[31m0.563\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m9\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[37m1.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.585\u001b[0m | \u001b[1m\u001b[31m0.560\u001b[0m | \u001b[37m1635014422_RNN-1_Dense-2-softmax\u001b[0m |\n",
"+-------+----------+----------+----------+-------+--------+-------+----------------------------------+\n"
]
}
],
"source": [
"model.train(epochs=10, init_logs=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You may already note that results are virtually identical to just using a basic Perceptron, although slightly better."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABPaklEQVR4nO3dd3xUZfb48c/JTHpvQEJLQHpCQlexoKjIKqg0C+rirvhT14Yra13Xvn7tunZd29pBRUDswAKrohAQQlOaQhIgCWRSSM/z++NOhkkyCQEymQTO+/WaV2ZuPTNJ5tznee49V4wxKKWUUvX5+ToApZRSbZMmCKWUUh5pglBKKeWRJgillFIeaYJQSinlkSYIpZRSHmmCUEop5ZEmCKVUmyciF4jIDhEpFpFBvo7nWKEJoh0QketEZIWIlIvIG/XmhYjI8yKSJyIOEVniNm+GiGwVkUIRyRaRJ0XE7jZ/u4iUOv/pikXkq3rbniEiu5zrvyYigW7z7heRtSJSJSL3NBH7ayJiROQ4D/N6iUiZiLztNu0cEVkmIgXOfb8qIuFu8x8TkV9FpEhENorI5c34/JKcMdS+z+0iclu9ZbaLyB4RCXWbdqWILHZ7bZzv2c9t2gP1fyf1tjtFRL4Tkf3u2zrEeHeLyHwRObM567emw/l9HKbHgOuMMWHAPufnYz/YSurIaIJoH7KBB4DXPMx7GYgB+jl/znCbNxcYbIyJAFKANOCGeuuPM8aEOR9n1U4UkTHAbcBooDvQA7jXbb3NwN+AzxoLWkROAno28b6eA36qNy0S670mOt9TZ+BRt/klwDjncn8EnhaRE5vYh7so5xfMJODvHr5wbcCNB9lGInBRM/cHsBd4Cnj4ENapVRtvGvA18ImITDuM7XjTkfw+DkV3YJ0XtquaoAmiHTDGfGyMmQPku08Xkb7AeOAqY0yuMabaGLPSbb0txpiC2sWBGqDBkXwj/gj82xizzhizD7gfmOa27TeNMZ8DRZ5Wdh7d/Qu4vpH5FwEFwLf13uu7xpgvjDH7nft9BRjpNv8fxpiNxpgaY8xyYClwQjPfU+02VmB92aTXm/UocIuIRDWx+iPAvc09ejXGfGOM+RAryR8WY8wuY8zTwD3A/9W2YEQkUUQ+EpFcEdkmIq7kLyL3iMiHIvKW8+h+nYgMdZt/q4hkOedtEpHRzul+InKbiGwRkXznNmKaiO2Qfh8iMs3Zqi1yxjzVbb93ichvzpbcWyISKSKBIlKMlbx/FpEtQG0rucDZwjrBud3/OVvJBc59nOicvsO5zT+6xXGOiKxyto53uLeCReRCZ2wRztdjna3Z+Ob+zo4WmiDat+HAb1hfWHnO7o+J7guIyCUiUgjkYR2JvlRvG+84v2C+EpE0t+kDgJ/dXv8MdBSR2GbGNgNYYoxZU3+G8x/vPuDmZmznFBo5chSRYGBYY/MbIyLHY7WoNtebtQJYDNzSxOofA4W4JctW9DHQAejjTBLzsH4vnbFaejc5W361xgPvA1FYrclnAUSkD3AdMMwYEw6MAbY717keOB84Fau1tA+rpXdQB/t9OLvvngHGOvd7IrDaOXua83EaVms1DHjWGFPubEUBpBljemL9TYCzhWWM+d75egSwBogF3nW+92FYB0WXAs+KSO22SoDLnZ/NOcA1InI+gDHmA+A74Bnn3/u/gSuNMbnN+RyOJpog2rcuWF90Dqx/5uuAN0WkX+0CziPyCKA38CKw2239qUASVvN9EfCl29FzmHO7tWqfh3MQItIV+H/A3Y0scj9W62TnQbZzJlZLprHtvIj1BfnlwWJyyhORUuB74Hlgjodl7gaub+Jo0QB/x+qiCmjmfltKbSskBuuLL94Yc58xpsIYsxWrteXe/bXMGLPAGFMN/AfrAAGgGggE+ouIvzFmuzFmi3Pe1cCdxpidxphyrFbLpGa2mJrz+6gBUkQk2BiTY4ypTSZTgSeMMVuNMcXA7cBFhzjOsM0Y87rz/X4AdAXucyaZr4AKnC1oY8xiY8xaZ8tnDfAeVlKs9RfgdKwDhnnGmPmHEMdRQxNE+1YKVAIPOL8k/ov1RX9W/QWNMb9iHdk97zbtf8aYUmd3zj+xunxOds4uBiLcNlH73GOXUj1PYf1jOurPEJF04AzgyaY24DzKfxeYZIz5xcP8R7GS4xTT/JLEcViJ76/AKMC//gLGmExgPtb4i0fGmAXATqwk6B7Ti24Dy3c0M6ZD0dn5cy9WUk90dqcUiEgBcAfQ0W35XW7P9wNBImI3xmwGbsL68t8jIu+LSKJzue5YYx2129yAlVA6NvX+PP0+6i9vjCkBLsRKQjki8pmzmxSsA5zf3Db5G2Cv934Oxv3gpxTAGFN/WpgzthEissjZenY4Y4qrXdDZNTvL+Z4eP4QYjiqaINq3Bt03WEe4jbHT9KCxwRqrACuZuHc5pQG7jTH5DdZqaDTwqLPftvZL6nsRuQTrizkJ+N057xZgoohk1K4s1mmMc4E/GWPqjFE4598LjAXOMsYUNiMeF+c4zRNAGXBtI4v9A5jOgS9kT+7E+kIOcdv21W4D/g8dSlzNdAGwB9gE7MA6Yo5ye4QbY/7QnA05W5YnYSUEA/yfc9YOrC4g9+0GGWOyGnt/jf0+PC1vjPnSGHMmkABsxGr1gNU66u4WYjegirpf+q5NN+c9HsS7WH9jXY0xkVitn9q//doDmT9htSyeaYH9tUuaINoBEbGLSBDWQJ1NRIKcTe8lwO/A7c5lRmL14X7pXO9KEengfN4fq9n+rfN1NxEZKSIBzu3NxDqC+p9zt28BfxaR/s5up7uAN9xi8nfG5AfYnduwOWf3xkoo6RwYCB4HfIJ11lVPt3kvYp0JNca53RTgC+B6Y8w8D5/F7cAlwBnNTFaNeRj4m/M91OE8wv6Ahmd8uS+zGMjE6gJrlIjYnPuwA37Oz6lBy+Ug2+goItdhJa7bjTE1wI9AkViDzcHO/aSIyLBmbK+PiJwu1mnLZVhH1jXO2S8CD4pId+ey8SJyXhPbavbvw/k+znOORZRjtVJr9/seMENEkp3jBA8BHxhjqjxsKte5Xo+DvdcmhAN7jTFlIjLc+R5q4wwC3sY6ALgC6CwijR1MHN2MMfpo4w+srgBT73GPc94ArD71EmA9cIHbeq9jHYGVYA1CPgoEua23xjkvHytxDK2335ud6xc6txXoNu8NDzFNayR+AxzXxHt7u17MNVhfHrWPdfW2VV5v/h0H+fySnOvZ3aYJVivpeufr7VhfcrXzu2J9eS5u7H1gDYoa4I0m9j3Nw+fU6PL14i12/n72AAuAs+stl4j1xboLazD5h9r34OFzdX0GwECcCQaru2o+kOhczs/5e9/knL8FeKiJWJv9+8BqNfwXazyrAKt/v7/bfu/GasHkYn1BRzfx2d/nXK4AON75OS9zm38cYOrtfydwkvP5JKxurCLn+3+29vPC6v783G29NOfn1MvX3wWt/RDnB6CUUkrVoV1MSimlPNIEoY4KIjLV7YwZ90ebvPq2vcWrjk3axaSUUsqjo6bYVVxcnElKSvJ1GEop1a6sXLkyzxjj8cLQoyZBJCUlsWLFCl+HoZRS7YqI/NbYPB2DUEop5ZEmCKWUUh5pglBKKeWRJgillFIeaYJQSinlkSYIpZRSHmmCUEop5dFRcx1Eu1dZCstfhIr9vo5EKdXeRCTC0CtafLOaINqKTZ/DN/c4X0hTSyqlVF1dhmqCOKplZ4AtAG7PAntr3+pYKaUa0jGItiJrFXRK1eSglGozNEG0BTXVkLMaEgf7OhKllHLRBNEW5P0KFcWQOMjXkSillIsmiLYgO8P62VlbEEqptkMTRFuQlQH+oRDX29eRKKWUiyaItiB7FSSmg5/N15EopZSLVxOEiJwtIptEZLOI3OZh/pMistr5+EVECtzmVbvNm+vNOH2qqgJ2rdXxB6VUm+O16yBExAY8B5wJ7AR+EpG5xpj1tcsYY2a4LX894P4tWWqMSfdWfG3GnvVQXa7jD0qpNsebLYjhwGZjzFZjTAXwPnBeE8tfDLznxXjaptoBaj3FVSnVxngzQXQGdri93umc1oCIdAeSgYVuk4NEZIWI/CAi53stSl/LyoDgaIhO8nUkSilVR1sptXERMNsYU+02rbsxJktEegALRWStMWaL+0oichVwFUC3bt1aL9qWlL3KGn8Qrb+klGpbvNmCyAK6ur3u4pzmyUXU614yxmQ5f24FFlN3fKJ2mZeNMUONMUPj4+NbIubWVbEf9mzQ7iWlVJvkzQTxE9BLRJJFJAArCTQ4G0lE+gLRwPdu06JFJND5PA4YCayvv267t2stmGodoFZKtUle62IyxlSJyHXAl4ANeM0Ys05E7gNWGGNqk8VFwPvGGOO2ej/gJRGpwUpiD7uf/XTU0AFqpVQb5tUxCGPMAmBBvWl313t9j4f1vgNSvRlbm5CVAeEJEJHg60iUUqoBvZLal7IztPWglGqzNEH4SmkB5G+GznoFtVKqbdIE4Ss5q62fWmJDKdVGaYLwlSwdoFZKtW2aIHwle5V19XRIjK8jUUopjzRB+Er2Km09KKXaNE0QvlCcC44deoGcUqpNayu1mI4tjVwgV1hWSXlljQ8CUkq1Z/42ISokoMW3qwnCF7IyAIGENNektTsdjH9uGXWuJ1dKqWZI7xrFnL+MbPHtaoLwhewMiO8DgWGuScu35WMM/P3c/gTYtedPKdV8caEt33oATRCtzxirBdHrrDqTM7McJEQG8eeTkn0UmFJK1aWHqq3NsRP25zUYoF6b5WBAYqSPglJKqYY0QbQ2DwPUJeVVbM0rIbWzJgilVNuhCaK1ZWWAnz90SnFN2pBTiDGQ0jnCh4EppVRdmiBaW3YGdBwA9kDXpMwsBwAp2oJQSrUhmiBaU00NZK/2MP5QSFxYIB3CAz2vp5RSPqAJojXt3QLlhQ0quK7LdpDaOQIR8VFgSinVkCaI1uShgmtZZTW/7inW7iWlVJujCaI1Za8CezDE93VN2ririOoao6e4KqXaHE0QrSk7wyqvYTtwfeJa1wC1nsGklGpbNEG0luoqyFnTYIB6XZaD6BB/OkcF+ygwpZTyTBNEa8ndAFWlDSq4ZmY7SOkcqQPUSqk2RxNEa6kdoHZrQVRU1bBpV5GOPyil2iRNEK0lOwMCIyH6QDG+X3YXUVlttMSGUqpN0gTRWrIyIDEd/A585Jk6QK2UasM0QbSGyjLYs77BAHVmtoPwIDvdYkJ8FJhSSjVOE0Rr2J0JNVUNB6izChmQqFdQK6XaJk0QrcHDAHVVdQ0bcgp1/EEp1WZpgmgN2RkQ2gEiOrsmbc4tpryqRktsKKXaLE0QrSErw2o9uHUlZWYVAugprkqpNksThLeVF0HeLw0quGZmOQgJsJEcF+qjwJRSqmmaILwtezVgPAxQOxiQGIHNTweolVJtkyYIb8teZf10G6CurjGszynU7iWlVJumCcLbsjMgshuExrkmbcsrYX9FtQ5QK6XaNE0Q3paVAZ0bjj+AXkGtlGrbNEF4U0k+FPzmcfwh0O7HcfFhPgpMKaUOThOEN3kYfwCrxEa/hAjsNv34lVJtl1e/oUTkbBHZJCKbReQ2D/OfFJHVzscvIlLgNu+PIvKr8/FHb8bpNdnOK6gT0l2TamoM67IKtXtJKdXm2Q++yOERERvwHHAmsBP4SUTmGmPW1y5jjJnhtvz1wCDn8xjgH8BQwAArnevu81a8XpGVAbG9IOhAMvh9736KyqtI0TOYlFJtnDdbEMOBzcaYrcaYCuB94Lwmlr8YeM/5fAzwtTFmrzMpfA2c7cVYvSN7lcfuJUDPYFJKtXneTBCdgR1ur3c6pzUgIt2BZGDhoawrIleJyAoRWZGbm9siQbeYwmwo3uWxgqu/TejdMdxHgSmlVPO0lVHSi4DZxpjqQ1nJGPOyMWaoMWZofHy8l0I7TB4quAKsy3bQp1M4Afa28tErpZRn3vyWygK6ur3u4pzmyUUc6F461HXbpuwM8LNDp1TXJGMMa7McOv6glGoXvJkgfgJ6iUiyiARgJYG59RcSkb5ANPC92+QvgbNEJFpEooGznNPaj6wM6NAP/IMPTCoopWB/pY4/KKXaBa8lCGNMFXAd1hf7BuBDY8w6EblPRMa7LXoR8L4xxrituxe4HyvJ/ATc55zWPhhjDVB7GH8AHaBWSrUPXjvNFcAYswBYUG/a3fVe39PIuq8Br3ktOG/auxXKChqU+F6X7cDmJ/TtpAPUSqm2T0dKvaGRK6jXZjno1SGMIH+bD4JSSqlDownCG7JXgT0IOvR3TTLGkJnl0O4lpVS7oQnCG7IyrLOXbP6uSXuKyskrriAlUUtsKKXaB00QLa2mGnJ+9ljBFXSAWinVfmiCaGm5m6CyxOP4gwj01xaEUqqd8OpZTMek2gquHk5x7RkfRkiAfuSq/amsrGTnzp2UlZX5OhR1mIKCgujSpQv+/v4HX9hJv61aWlYGBIRD7HF1Jq/LdjAiOcZHQSl1ZHbu3El4eDhJSUmIiK/DUYfIGEN+fj47d+4kOTm52etpF1NLy14Fiengd+CjzSsuJ8dRpuMPqt0qKysjNjZWk0M7JSLExsYecgtQE0RLqqqA3ZkNLpDTAWp1NNDk0L4dzu9PE0RL2p0J1RUeKrhaJTZ0gFqplnHPPffw2GOPAfDGG2+QnZ19yNt48cUXeeutt1o6tKOKjkG0pEYHqB0kxYYQEdT8wSGlVPO88cYbpKSkkJiY2GBedXU1NpvnygVXX321t0M7bFVVVdjtvv961hZES8paBSGxENWtzuTMbAcDtHtJqSPy4IMP0rt3b0466SQ2bdoEwOzZs1mxYgVTp04lPT2d0tJSkpKSuPXWWxk8eDCzZs3ilVdeYdiwYaSlpTFx4kT2798P1G2FjBo1iltvvZXhw4fTu3dvli5d2mD/xcXFjB49msGDB5Oamsqnn37qmvfWW28xcOBA0tLSuOyyywDYvXs3F1xwAWlpaaSlpfHdd9+xfft2UlJSXOs99thj3HPPPa4YbrrpJoYOHcrTTz/NvHnzGDFiBIMGDeKMM85g9+7drjiuuOIKUlNTGThwIB999BGvvfYaN910k2u7r7zyCjNmuO7ofNh8n6KOJtkZVuvBra+vYH8FO/aWMnVEdx8GplTLuXfeOtY7u01bSv/ECP4xbkCj81euXMn777/P6tWrqaqqYvDgwQwZMoRJkybx7LPP8thjjzF06FDX8rGxsWRkWC36/Px8pk+fDsBdd93Fv//9b66//voG+6iqquLHH39kwYIF3HvvvXzzzTd15gcFBfHJJ58QERFBXl4exx9/POPHj2f9+vU88MADfPfdd8TFxbF3r1V4+oYbbuDUU0/lk08+obq6muLiYvbt29fk51BRUcGKFSsA2LdvHz/88AMiwquvvsojjzzC448/zv33309kZCRr1651Lefv78+DDz7Io48+ir+/P6+//jovvfTSwT72g9IE0VIqSiB3I/Q9t87k2vEHvUmQUodv6dKlXHDBBYSEhAAwfvz4Jpe/8MILXc8zMzO56667KCgooLi4mDFjxnhcZ8KECQAMGTKE7du3N5hvjOGOO+5gyZIl+Pn5kZWVxe7du1m4cCGTJ08mLi4OgJgY63T2hQsXusY4bDYbkZGRB00Q7nHv3LmTCy+8kJycHCoqKlynp37zzTe8//77ruWio6MBOP3005k/fz79+vWjsrKS1NRUjtQxnyCMMTz/8/NM7DWRTqGdDn9DOWvA1DQYoK49g2mADlCro0RTR/ptRWhoqOv5tGnTmDNnDmlpabzxxhssXrzY4zqBgYGA9WVeVVXVYP4777xDbm4uK1euxN/fn6SkpEM+bdRut1NTU+N6XX9997ivv/56br75ZsaPH8/ixYtdXVGNufLKK3nooYfo27cvV1xxxSHF1Zhjfgxie+F23lr3FpPmTWLxjsWHv6HGBqizC+kcFUx0aMDhb1upY9wpp5zCnDlzKC0tpaioiHnz5rnmhYeHU1RU1Oi6RUVFJCQkUFlZyTvvvHPYMTgcDjp06IC/vz+LFi3it99+A6wj91mzZpGfnw/g6mIaPXo0L7zwAmANljscDjp27MiePXvIz8+nvLyc+fPnN7m/zp07A/Dmm2+6pp955pk899xzrte1rZIRI0awY8cO3n33XS6++OLDfp/ujvkEkRyZzIfjPiQxNJHrF17PIz89QmV15aFvKCsDIjpDeMc6kzOzHKTqALVSR2Tw4MFceOGFpKWlMXbsWIYNG+aaN23aNK6++mrXIHV9999/PyNGjGDkyJH07dv3sGOYOnUqK1asIDU1lbfeesu1rQEDBnDnnXdy6qmnkpaWxs033wzA008/zaJFi0hNTWXIkCGsX78ef39/7r77boYPH86ZZ57ZZDz33HMPkydPZsiQIa7uK7DGUfbt20dKSgppaWksWrTINW/KlCmMHDnS1e10pMTtTp/t2tChQ03t4M7hqKiu4PEVj/PuxncZEDuAR095lK4RXZu/gWcGWfd/uOjAEUpRWSWp93zFLWf15rrTex12bEr52oYNG+jXr5+vw1AHce655zJjxgxGjx7tcb6n36OIrDTGDPW0/DHfgqgVYAvg9hG389RpT/F70e9MmT+FL7Z/0byVS/dZtxmtN/5Qe6aHnuKqlPKmgoICevfuTXBwcKPJ4XAc84PU9Y3uNpp+Mf3425K/MfO/M/kx50f+NuxvBNmDGl+p9hajHsYfQM9gUkp5V1RUFL/88kuLb1dbEB4khiXy+tmv8+eUPzPrl1lcsuASthZsbXyFrNoB6vQ6kzOzHHSKCCI+PNB7wSqllJdogmiEv58/Nw25iRfPeJH80nwu+uwi5myeg8cxm+xVENMDgusODFn3oNbTW5VS7ZMmiIMY2Xkks8bNIjUulb//7+/csewOSipL6i6UvapB99L+iiq25BYzQLuXlFLtlCaIZugQ0oGXz3yZa9OvZcG2BVw0/yI27t1ozSzaDYVZDQaoN+QUUWO0xLdSqv3SBNFMNj8b16Rdw6tnvcr+yv1M/Wwq7218D5O10lrAQwVXQK+BUMoL3AvtHapp06Yxe/bsFo7o6NTsBCEiId4MpL0Y1mkYs8bPYkTCCB5a/hA3r/kXhTYbJAyss1xmloO4sAA6RugAtVLqAE9lPNqqgyYIETlRRNYDG52v00Tkea9H1obFBMXw7OhnuWXoLSwu3cnkLl342bG5zjKZ2YUMSIzUu3Ap1UI8lfveuHEjw4cPdy2zfft2V5G6++67j2HDhpGSksJVV13l+QQTN42VBfdUths8l/iu3zoJCwsDYPHixZx88smMHz+e/v37A3D++eczZMgQBgwYwMsvv+xa54svvmDw4MGkpaUxevRoampq6NWrF7m5uQDU1NRw3HHHuV57U3Oug3gSGAPMBTDG/Cwip3g1qnbAT/z4Y//LGfTlffwtPpppn0/jhsE38McBf6SiyvDr7iJO7xvv6zCVanmf3wa71rbsNjulwtiHG53dWLnvvn37UlFRwbZt20hOTuaDDz5wVUS97rrruPvuuwG47LLLmD9/PuPGjWt0HxMmTPBYFtxT2e5169Z5LPHdlIyMDDIzM11VWV977TViYmIoLS1l2LBhTJw4kZqaGqZPn86SJUtITk5m7969+Pn5cemll/LOO+9w00038c0335CWlkZ8vPe/X5rVxWSM2VFvUrUXYml/Cn5nYGEeH/a6gtO6ncYTK5/gL9/+hZ9+/52qGqPjD0q1EPdy3xEREXXKfU+ZMoUPPvgAoE6CWLRoESNGjCA1NZWFCxeybt26JveRmZnJySefTGpqKu+8845r+YULF3LNNdcAB8p2N1biuynDhw93JQeAZ555hrS0NI4//nh27NjBr7/+yg8//MApp5ziWq52u3/6059cpcNfe+21FqvWejDNaUHsEJETASMi/sCNwAbvhtW6TGUl4n8YtwN1VnCN6HoCjydey4ebPuSRnx5h9e5p2EImMiDxtBaOVKk2oIkjfV+48MILmTx5MhMmTEBE6NWrF2VlZVx77bWsWLGCrl27cs899xy0NHdzy4I3xb2cd01NDRUVFa557qW8Fy9ezDfffMP3339PSEgIo0aNajK+rl270rFjRxYuXMiPP/54RFVpD0VzWhBXA38BOgNZQLrz9VGhat8+to4bT8Hs2Qfto2wgKwNsAdAxBRHhwr4X8u4572KqAwnp9irzfn+D6hptbCl1pJoq992zZ09sNhv333+/q/VQ+2UbFxdHcXFxs85aaqwsuKey3Y2V+E5KSmLlSuvMxrlz51JZ6bkytMPhIDo6mpCQEDZu3MgPP/wAwPHHH8+SJUvYtm1bne2Cdb+HSy+9lMmTJzd6n+2WdtAEYYzJM8ZMNcZ0NMZ0MMZcaozJb43gWoUx2BM6kXPX38m5/Q5qnANTzZK9CjqmgP3AvR76xPQh1vE3os3xvPDzC0z/ejp79u/xQuBKHTuaKvcNVivi7bffZsqUKYBVm2j69OmkpKQwZsyYBst70lhZcE9luxsr8T19+nT++9//kpaWxvfff1+n1eDu7LPPpqqqin79+nHbbbdx/PHHAxAfH8/LL7/MhAkTSEtLq3OHufHjx7vuR91aDlruW0ReBxosZIz5k7eCOhxHUu7bVFeT9/wL5D3/PIHH9aTzU08R2LNn0yvV1MDD3SDtQjjncdfkiqoaUv7xJVeclET/Xr/w4PIHCbIF8dDJD3FS55MOKz6lfE3LffveihUrmDFjBkuXLj3sbXij3Pd84DPn41sgAig+7AjbILHZiL/+Orq++gpV+XvZNnkKjnmN3+kJgPxfoaKowQVyv+4poqK6hpTESM477jzeP+d94kLiuOaba3hixRNU1hzGzYiUUse0hx9+mIkTJ/LPf/6zVffbnC6mj9we7wBTAI/Zpr0LGzmS5E8+JqhfP7JnziTnnnuoKS/3vHBtBdd6JTbWZTlLfDvPYOoR1YN3//AuU3pP4fV1rzPti2lkFWd57T0opY4+t912G7/99hsnndS6vRCHU2qjF9ChpQNpK/w7dqT7G68Te+WfKXj/A7ZffDEVv//ecMHsVeAfCnG960xem+UgPNBO95gDF54H2YP4+wl/57FTH2NrwVYmz5vMN7994+23opRSR6Q5V1IXiUhh7U9gHnBrczYuImeLyCYR2SwitzWyzBQRWS8i60TkXbfp1SKy2vmY29w31BLE358Ot9xCl+efpzIrm20TJlL41Vd1F8rOgIQ08Kt7NkFmtoP+iRH4+TW8gnpM0hg+HPch3cK7MWPxDB784UHKqxtpoSillI81p4sp3BgT4faztzHmo4OtJyI24DlgLNAfuFhE+tdbphdwOzDSGDMAuMltdqkxJt35GI8PhJ9+GskffURAcjJZN9zI7n/+E1NRAdWV1pWk9bqXqqpr2JBT2GQF167hXfnP2P9wef/LeX/T+1y64FK2O7Z7+Z0opdShazRBiMjgph7N2PZwYLMxZqsxpgJ4Hziv3jLTgeeMMfsAjDFt7nzQgC6d6f7O20Rfeil733yL3y67nMq1S6CqDBIH1Vl2a14JZZU1B71JkL/Nn5nDZvLs6c+SU5LDlPlTmLdlXpPrKKVUa2uqBfF4E4/m1NntDLiX6NjpnOauN9BbRP4nIj+IyNlu84JEZIVz+vmediAiVzmXWeHNwlV+AQF0uutOOj/1JOWbN7Ptylsozg5s0IJYu/PQSnyf2vVUZo+bTb+Yftyx7A7uWnYXBWUFLR2+Ukcd93Lfb7zxBtnZ2Ye8jRdffNFVvqI5tm/fTnBwMOnp6aSlpXHiiSe6igYCfP755wwdOpT+/fszaNAg/vrXv7pi7dy5M+np6a5HQUHBIcfrC42W2jDGtEadCDvWoPcooAuwRERSjTEFQHdjTJaI9AAWishaY8yWejG+DLwM1nUQ3g424uyzCerbl51XTGbHklhi35hD/A03IHbrY8zMdhDsbyM5LqzZ2+wU2ol/j/k3L/z8Aq+seYVvf/+WywdczmX9LiMsoPnbUepY9cYbb5CSkkJiYmKDedXV1Y1edXz11Vcf8r569uzJ6tWrAXjppZd46KGHePPNN8nMzOS6667js88+o2/fvlRXV9ep0DpjxgxuueWWQ96frzXrLCYRSXEOJl9e+2jGallAV7fXXZzT3O0E5hpjKo0x24BfsBIGxpgs58+twGJgEG1AQFISSRMCiEqPJP/lV/j9ij9RucfqGVuXVUj/xAhsHgaom2L3s3P9oOv5aPxHjEgYwfOrn2fsx2N5c92blFU1XT9GqWOFp3Lfs2fPZsWKFUydOpX09HRKS0tJSkri1ltvZfDgwcyaNavRMt7urZBRo0Zx6623Mnz4cHr37t2si9EKCwuJjrbuQ//II49w5513uq7AttlsrgJ/7dlBi/WJyD+wjvD7AwuwBp2XAQdrm/0E9BKRZKzEcBFwSb1l5gAXA6+LSBxWl9NWEYkG9htjyp3TRwKPNPM9eVdlKX57N5Aw/SaCiway69772HbBBBIefZR12Q4mDely2JvuFd2Lp057isy8TP616l88tuIx3lr3FlcNvIoJvSbgbzuMgoJKtbD/+/H/Dtxyt4X0jenLrcMbPzmysXLfkyZN4tlnn+Wxxx5j6NADl2fFxsaSkWFdq5Sfn++xjHd9VVVV/PjjjyxYsIB7772Xb75peCr6li1bSE9Pp6ioiP3797N8+XLAqgRb26XkyZNPPsnbb78NQHR0NIsWLWrGp+J7zWlBTAJGA7uMMVcAacBBO9mNMVXAdcCXWNVfPzTGrBOR+0Sk9qykL4F85w2JFgEznXWe+gErRORn5/SHjTHrD/G9eceutWCqIXEwUeefT/KHH2CLimLnlVcyfs0XpCSEH/EuUuJSeOnMl3htzGt0Du/MA8sfYNyccczdMleL/6ljUlPlvj1xr2HUWBnv+iZMmADAkCFD2L59u8dlaruYtmzZwlNPPcVVV13VrPhnzJjB6tWrWb16dbtJDtC8ct9lxpgaEakSkQhgD3W7jhpljFmA1epwn3a323MD3Ox8uC/zHZDanH20unpXUAf26kXyhx/w0423cvmyL+FfDqqefhx7M+rDH8ywTsN48+w3WZq1lGdXPcudy+7ktbWv8ZdBf+GMbmfo3eqUTzR1pN9WuBfJa24Z78BA6/bANputWbcFHT9+vKtw3oABA1i5ciVpaWlHHnwb0tRprs+JyEnAjyISBbwCrAQygO9bJ7w2KDsDwjpBxIEBMb/QUL694FqeGzwZ+TmDbRdMYL+zeXukRIRTupzC++e+z+OnPk4NNdy8+GYu+uwilmUtO/QS5Uq1Q02V+w4PD6eoqKjRdRsr432kli1bRk9nUc+ZM2fy0EMP8csvvwDWvSBefPHFFtuXrzTVgvgFeBRIBEqA94AzgQhjzJpWiK1tyspocHorWPegLjnxLJLuuoidN83gt8sup8PNNxPzpyta5EjfT/w4K+ksTu92Op9t/YwXfn6Ba765hsEdBnPD4BsY0nHIEe9DqbbKvdx3hw4d6pTvnjZtGldffTXBwcF8/33DY9faMt7x8fGMGDGiyWRyMLVjEMYYAgICePXVVwEYOHAgTz31FBdffDH79+9HRDj33HNd67mPQQDMmTOHpKSkw46jtTSn3Hd3rAHmi4BgrETxrjHmV++H13xHUu672cocVonv0+6CU2e6JhtjSLv3K8alJfLgBalUFxWRc8edFH39NWGnn07iPx/CFtmytx+trK7ko18/4qU1L5FXmsfIxJFcP/h6BsQOaNH9KAVa7vto0eLlvo0xvxlj/s8YMwjrjKPzgZY9haG9yF5t/exc94zbHXtLKSyrcpXYsIWH0/mZp+l4x+0UL1nCtgkTKV2b2aKh+Nv8uajvRSyYsICbh9xMZn4mF82/iBmLZrClYMvBN6CUUgfRnGJ9dhEZJyLvAJ8Dm4AJXo+sLcpeZf1MqJsgMrOtK6hTEg+0EkSEmMsvJ+nt/2BqavjtkkvY+847LT5mEGwP5oqUK/h8wudck3YN3+d8zwWfXsAdS+9gR9GOg29AKaUa0dQg9Zki8hrWxWzTsW4Y1NMYc5Ex5tPWCrBNyc6AqO4QGltncmaWA3+b0LtTwyufg9PTSf74I0JOPIHd9z9A9l//SnVxSYuHFh4QzrXp1/L5hM+ZNmAaX/32FeM/Gc/939/P7pLdLb4/pdTRr6kWxO3Ad0A/Y8x4Y8y7xpiW/2ZrT7JWeRygXpvloHfHcALtni/pt0dH0/WFF4i/+WYKv/iS7ZMmUeZWw6UlRQdFc/PQm1kwYQETe0/k418/5pxPzuGxnx5jX9k+r+xTKXV0ajRBGGNON8a8Wltp9ZhXkgeO3xvcYtQYw7rswjrdS56Inx9xV02n2xuvU1NSwvYpF1Lw0UGrph+2DiEduOv4u5h3wTzGJI3hPxv+w9kfnc1zq5+jqOLwz+JQSh07DueOcsemRm4xmuMoY29JxUFLfNcKHT6c5E8+JnjQIHLuvIvs2++gprS0paN16RLehQdPepBPxn/CyM4jefHnFxn78Vhey3yN0irv7Vcp1f5pgmiu7AxArLvIucnMsgaoBzSzxDeAPS6Obv9+lbhrr8UxZw7bp1xI+datLRltAz2ievDEqCf44NwPSI1L5cmVT/KHj//Aexvfo7K60qv7VqqluRfaO1TTpk1j9uzZHqcnJyeTnp5O3759uffee13ziouL+X//7//Rs2dPhgwZwqhRo1x1mGw2W51S3g8//PDhvak2qDmlNhRYLYj4PhBYt9ZSZpYDm5/QP6F5LYhaYrMRf8P1BA8eTPbMmWybNJmEe+8lcty5B1/5CPSP7c8LZ7xAxu4Mnln1DA8tf4g3Mt/g6rSrGddzHHY//ZNQx65HH32USZMmUVZWRv/+/bn88stJTk7myiuvJDk5mV9//RU/Pz+2bdvG+vVWebjg4GBXCfCjjbYgmsMY6xTXxIYVxzOzCzkuPowgf88D1AcTdtJIkj/5mKC+fcmeOZPfr5xO2UbvX2YyuONgXh/zOi+d8RJRQVHc/d3dXPDpBXyx/QtqTI3X96/UofJU7nvjxo0MHz7ctcz27dtJTbXKuN13330MGzaMlJQUrrrqqkM6xbyszCqzHxoaypYtW1i+fDkPPPAAfn7WV2ZycjLnnHNOS721NksPF5ujMAtK9jQYoAarBXFSr7gj2rx/p050f/MN9v7nbfJeeoltF0wgcvx44m+8AX8PN0FpKSLCiZ1P5ITEE1j4+0L+tepfzPzvTF6OfpmJvSYyNnksMUFHXnRQHV12PfQQ5Rta9iAmsF9fOt1xR6PzGyv33bdvXyoqKti2bRvJycl88MEHrkqu1113HXffbdUGveyyy5g/fz7jxo1rMo6ZM2fywAMPsHnzZm644QY6dOjADz/8QHp6eqM3HiotLSU9Pd31+vbbb69TTbY90xZEczQyQL2nsIw9ReUHPYOpOcTfn9g/XcFxX31JzJ+uoPDzz9ly9lh2P/Io1Q7HEW+/yX2LMLr7aD4a/xEPnfQQdrHz8I8PM/rD0Vy/8Hq++e0bKqorvBqDUk1pqtz3lClT+OCDDwDqJIhFixYxYsQIUlNTWbhwYaNlvt09+uijrF69ml27dvHtt9/y3XffHXSd2i6m2sfRkhxAWxDNk50BfnbomFJncu0V1KldWq7Oki0yko4zZxIzdSq5z/yLva+/TsFHHxF31VVEXzoVP2dJYm+w+dkY13Mc43qO45d9vzB/y3zmb53P4h2LiQyM5OyksxnfczypcalaavwY1tSRvi9ceOGFTJ48mQkTJiAi9OrVi7KyMq699lpWrFhB165dueeee1zdRs0RFhbGqFGjWLZsGRMnTuTnn39u8valRyttQTRHVgZ0HAD+QXUmZ2YVIgL9DnGAujn8ExNJfPif1imxAwey59FH2TJ2LI5PP8XUeH+MoHd0b24eejNfTfqKF894kRMTT2TO5jlMXTCV8XPG88qaV8gpzvF6HEpB0+W+e/bsic1m4/7773cdvdcmg7i4OIqLiz2etdSUqqoqli9fTs+ePenZsydDhw7lH//4h2scY/v27Xz22Wct9O7aLk0QB1NTYxXpa2T8ITkulLBA7zXEgvr2pdsrL9Pt9dewR0WTfettbJswkeJl//PaPt3Z/eyM7DySR055hMVTFnPvifcSGxzLM6ueYcxHY/jzl3/m082fsr9yf6vEo45N7uW+x44dW6fcN1itiLfffpspU6YAEBUVxfTp00lJSWHMmDENlm/MzJkzSU9PZ+DAgaSmprruMvfqq6+ye/dujjvuOFJSUpg2bRodOnQADoxB1D5uu+22FnznvnXQct/thdfKfedthmeHwPh/weDL68w68Z/fMiw5hqcvanh2kzeYmhoKF3xO7pNPUpmVReiJJ9DhllsI6t+/VfbvbmfRTuZtnce8LfPYUbSDYHswZ3Q7g3E9xzG803BsfsdWU/xop+W+jw6HWu5bxyAOpraCa71TXPOLy8l2lLXIAHVziZ8fkeeeQ/hZZ1Lw3nvkPf8C2yZMJGLcOOJvvJGALp1bLZYu4V24Ju0arh54NatzVzN3y1y+3PYl87bOo2NIR87tcS7je46nR1SPVotJKdWytIvpYLIzwB4M8XWz7rrsQgAGNLPERkvyCwgg5o9/pOfXXxE7fTpFX33F1rFj2f3w/1G1r3VLZ4kIgzoM4h8n/IOFUxby6KmP0iemD2+se4PzPj2Pi+dfzLsb3qWgrKBV41JKHTlNEAeTlQEJA8FWt7FVewbTgFZsQdRni4igw19vpueXXxAxfhx733qLLWeNIf/VV6k5hDM2WkqQPYizk87mudHP8c3kb5g5dCaVNZX888d/ctqs07hx4Y18+9u3WtpDqXZCE0RTqqsg5+dGB6i7x4YQGezvg8Dq8u/UicQHHyR5zieEDB7MnsceZ8vZYyn4+BNMdbVPYooLjuPyAZcze/xsZo+bzSV9L+Hn3J+5afFNnDbrNB784UEy8zJb/AZKynv0d9W+Hc7vTxNEU3I3QlWpx3tAZGYdvMR3awvq3ZuuL71ItzffxB4fT84dd7DtggkUL1ni03/uPjF9mDlsJt9M/obnRz/PCQkn8PGvH3PxZxdz3qfn8eraV9lVsstn8amDCwoKIj8/X5NEO2WMIT8/n6CgoIMv7EbPYmpKxlsw93q4biXEHeea7NhfSdp9X/G3s/tw7ajjmtiA7xhjKPriC/Y8+RSVv/9OyIgRdLjlFoJTUw6+cisorCjkq+1fMW/LPDL2ZCAIwxOGc17P8xjdbTQh/iG+DlG5qaysZOfOnYd0sZlqW4KCgujSpQv+/nV7PfQspsOVlQGBkRBT90ycdTkN70Hd1ogIEWPHEj56NPs++JC8559n++TJRPzhD8TPuImArl19Gl9EQASTek9iUu9J7Cjcwbyt85i7ZS53LLuDYHswZ3Y/k7HJYxnScQjB9mCfxqrA39+f5ORkX4ehWpm2IJry0qkQFAl/nFtn8stLtvDQgo1k/P1MYkIDWnafXlJdXEz+q6+y9403MdXVRF98EXHXXIM9OtrXobkYY8jYk8G8LfP4cvuXFFcW4+/nT1p8GiMSRnB8wvEMiBuAv5/vx32UOlo01YLQBNGYqnJ4qDOc8Bc48946s254bxUrf9vH/247veX210oqd+8h79lnKfjoI/xCQoidPp2Yyy/DL7htHaWXVZWxcvdKlucs54ecH9i4dyMGQ6h/KEM7DmVEwghGJIygV1QvrQul1BHQLqbDsSsTaio9D1BnOxiQ2PrXP7QE/44dSLj/PmL+eDl7nniS3CefZN+77xJ//XVEXnAB0kaKkQXZgxjZeSQjO48EoKCsgB93/cjynOUs37Wc/+78LwAxQTGu1sWIhBF0Dmu9iwWVOtppgmhMtrPEd71TXIvLq9iWV8L56e37iyjwuOPo+vxz7F+xgt2PPkrOXX9n75tvEn/zzYSNGtXmjsqjgqI4K+kszko6C4Cc4hx+yPmB5buWszxnOZ9v+xyALmFdOD7RShbDOw3X+1kodQQ0QTQmKwNC4yGyS53J67MLMQZSD+Ee1G1ZyNChJL3/PkVffU3uE0+w85prCR4yhJiplxB2xhn4BbTNMZaEsAQu6HUBF/S6AGMMWx1b+SHnB37I+YEvtn3B7F+s6p19ovu4WhhDOg7Rs6OUOgSaIBqTnWG1HuodSWdmOa+g9kGJDW8RESLGnEX46aexb9Ys9r76b7Ju/iu2qCgizzuPqCmTCezZ09dhNkpE6BnVk55RPZnabypVNVWsz19vtTBylvPexvd4a/1b2MXOwPiBroSRGpeKv00HvJVqjA5Se1JeBP/sCqNusx5ubv5wNct+zePHO89omX21QaamhpLvvqdg1iyKFi6EykqCBw8mavJkIs4e0+YGtA+mrKqMVXtWuQa81+evx2AItgczpOMQ1/hF7+je+IleO6qOLTpIfahy1gDGY4mNdVmFpBwl3UuNET8/wk4aSdhJI6nKz8cx51MKZs0i5/bb2f3gg0SMO5foyZN9Umb8cATZgzgh8QROSDwBAEe5gxW7VvB9zvcsz1nOY1mPARAdGM3whOFWC6PT8XQJ79LmxmKUak2aIDxxDVDXLfFdWlHNr3uKGJPSyQdB+YY9NpbYP/+JmD9dQemKFeybNQvHx59Q8N77BPXvT9SUyUScey62sDBfh9pskYGRjO4+mtHdRwOwq2SX6wypH7J/4MvtXwKQGJrIiIQRDOowiH6x/egZ2VO7pNQxRbuYPJl1Bez8CWZk1pmc8fs+Jjz/HS9fNoSzBhw7SaK+aocDx7z5FMyaRfmmTUhwMBFjxxI1aRLBg9Lb9VG3MYZthdus02lzlvPjrh8pqigCrLvr9YzsSZ+YPvSN6UvfmL70ielDRMDRMx6ljj3axXSosjMatB4A1jkHqI/2LqaDsUVGEnPpVKKnXkJZZiYFH86i8LPPcHz8MYG9jiNq0iQixo9vU1dpN5eI0COyBz0ie3Bx34uprqlmR9EONu7daD32beS77O+Yu+XA1fWdwzrTJ7qPK2H0i+lHp9BO7TpRKgVebkGIyNnA04ANeNUY87CHZaYA9wAG+NkYc4lz+h+Bu5yLPWCMebOpfbVYC2L/XngkGc64B06aUWfWrbPX8PWG3ay86wz956+nuriEws8XUDB7NmU/r0H8/Qk/80yipkwmZPhwxO/oGvzNK81j095NbNi7gU17N7Fx70Z+K/wNg/X/FBEQ4UoYfWP60ie6Dz2iemiZENXm+KQFISI24DngTGAn8JOIzDXGrHdbphdwOzDSGLNPRDo4p8cA/wCGYiWOlc51vX+7tEYukANYm+UgpXOkJgcPbGGhRE+eTPTkyZRt2kTBrNk45s6lcMEC/Lt1I2rSJCLPPw9/543e27u44DjiOse5rvQG2F+5n18Lfq2TOD7c9CHl1eUA+Pv5c1zUcQ0SR1hA+xm/UccWb3YxDQc2G2O2AojI+8B5wHq3ZaYDz9V+8Rtj9jinjwG+Nsbsda77NXA28J4X47Vk1d6DOr3O5PKqan7ZXcSoPvFeD6G9C+rTh0533UmHW/5K0ddfU/DhLHKfeILcp58m7LRRRE+eTOhJJ7WZsh4tJcQ/hLT4NNLi01zTqmqq+L3wd1f31Mb8jSzesZhPNn/iWqZreFdXsqhNHh1DOuqBiPI5byaIzsAOt9c7gRH1lukNICL/w+qGuscY80Uj6zaobSEiVwFXAXTr1q1los5eBbG9rCqubn7ZVUxVjTnmxx8OhV9QEJHjxhE5bhzl27ZRMHs2jk/mUPzNt9g7dSJqwgSiJk7Av3P7LlvSFLufnR5RPegR1YM/8AfAGgjPLc1l496Nru6pjXs38vVvX7vWiw6MPtDKiOlD3+i+dI/ormdRqVbl60FqO9ALGAV0AZaISGpzVzbGvAy8DNYYRItElJ0BSSc3mFx7D+q2fA+ItiwwOZmOM2fS4cYbKVq0mIJZs8h74QXyXniB0JNOImrSJMJPPw3xP/q/AEWEDiEd6BDSgVO6nOKaXlJZwi/7fqmTON7d8C4VNRUA+IkfnUI60SW8C13Du7p+1j7Xs6lUS/NmgsgC3O9K08U5zd1OYLkxphLYJiK/YCWMLKyk4b7uYq9FWqswB4pyPFZwXZvlICLITteY9nUVcVsjAQFEjDmLiDFnUbEzC8fHH1Pw8cdk3XgjtthYoi44n6hJkwhISvJ1qK0u1D+UQR0GMajDgTPoKmsq2e7Yzsa9G/m96Hd2FO1gR9EOFu1YxN6yvXXWjwyMpGtY1zrJo/Znh5AOepW4OmReO4tJROzAL8BorC/8n4BLjDHr3JY5G7jYGPNHEYkDVgHpOAemgdpv6gxgSO2YhCctchbTxs/g/UvgT19Bt7q9Yec9u4zQQDvvTj/+yPahGjDV1RQvXUrBrNkUL14M1dWEDBtG5PnnE3bqKdjj4nwdYptUUlnCzqKdrqTh/jynJIdqU+1aNtAWSOewznVaHbUJpEtYFwJsbbMoo/I+n5zFZIypEpHrgC+xxhdeM8asE5H7gBXGmLnOeWeJyHqgGphpjMl3Bn0/VlIBuK+p5NBisjJAbNCpbi9XZXUNG3YVMe3EJK+HcCwSm43wUaMIHzWKyt17cMyZQ8Hs2eTceScAQQMGEHbqKYSdcgpBqalH3eD24Qr1D6VPTB/6xPRpMK+yppJdxbvYUVw3cewo2sFPu36itKrUtaxgdXm5Jw731kdkoHarHqv0Smp3/7kAinPhmmV1Jm/IKWTs00t55uJBjE9LPLJ9qGYxNTWUbdhAydKlFP93CaU//ww1Ndiiogg96STCTjmZ0JNOwh6j93s4VMYY9pbtrdPy2Fl8IInklebVWT48IPxA0gjr4ho/iQuOIz4knvjgeG2BtGN6JXVzGGOdwdRvXINZa2uvoG6nd5Frj8TPj+ABAwgeMIC4q6+muqCA4v/9j5IlSyleupTC+fNBhKDUVMJOOYWwU04mKCXlqLsgzxtEhNjgWGKDY0nvkN5g/v7K/WQVZ9Vpdews2smG/A18+9u3VJmqButEBkYSHxxPXHDcgeQRHO9KIPHB8cSFxBFs1zG89kQTRK1926F0XyMVXB2EBthIig1t/bgUgHVvinPOIfKcc6zWxbr1FC/5LyVLlpL33HPkPfsstpgYQk8aSdgppxI68sR2WeqjLQjxD6FXdC96RfdqMK/G1LCvbB+5pbnk7s8lrzSPPfv3kFtqPc/dn8tPu34itzSXqpqGiSTcP5y4kLrJw1NSCfXX/7W2QBNErUYquAJkZhcyIDESPz+9cKktED8/glNTCE5NIf4vf6Fq3z5Klv2P4iVLKFm6jMK588DPj+CBAwk95WTCTjmVoP79tHXRAvzEz9X66BvTt9HljDE4yh3sKd1D3v48K6E4k0ptMlm9ZzV5pXmuK83dhdhDiA9xJo/gDh6TSmRgJJGBkVq+xIs0QdTKygBbIHQcUGdydY1hfXYhFw9voQvxVIuzR0cTOe5cIsedi6mupiwzk+IlSylesoS8fz1L3jP/whYXR1jt2MXIkdgideDVm0SEqKAoooKi6B3du9HljDEUVRa5Eof7z7xSK7Gsy19H7s7cOgPr7sL8w1zJIjIgkqjAKCICI4gKjCIyMNL1031+eEA4Nj892eFgNEHUyl5lnb1U70rVrbnFlFZWk3IU3WL0aCY2G8FpaQSnpRF//XVU5edTsmyZlTAWLcIxZ47VukhPt8YuTj2FwL59tayFj4gIEQERRARE0DOq6dvallSWsGf/HvJK88grzcNR7qCgvABHuePA8woH2SXZrmm1xRMb7BchPCDclUBcCaVegqmTXAIjCfcPP6b+VjRBANRUQ/ZqGDS1wSzXFdRaYqNdssfGEnneeUSedx6muprSNWusrqglS8l96ilyn3oKe3y81RV18imEjjwRW3i4r8NWHoT6h5IcmUxyZHKzlq8xNRRVFNVJJAXlBRRWFNZ57Sh3UFBWwHbHdgrLCymqLGp0mzaxEREQUSdphNpDCQ0IJcw/jBD/EML8w+o8D/UPJdT/wPxQ/1Dsfu3jq7d9ROlteb9AZYnHAerMrEKC/P3oEaeDZu2d2GyEDBpEyKBBcOONVOXmUrzsfxQv+S9FX3+D46OPwW4nJD2dUOd1F4G9ex9TR4xHEz/xc32Jd6P5XcRVNVWuJFJYbv10b6m4t1Zy9+eyvXI7xZXF7K/cT1l1WbP2EWwPJsQeQljAgQRSm0TqJ5QGCScglFB7KGEBYQTbg716hbwmCLDGH8BjiY3MLAf9EyKw23SA82hjj4+3SntccD6mqspqXfx3CcVLl5D7+BPkPv4E9o4dCT1+BEGpAwkemEpg3774Beg5/0czu5+dmKAYYoIO/RqbyppK9lfup6SyxJU0iiuLDzyvKKakssQ1v/Z5SWUJ2cXZdeZ5OgusPkEI8Q8hvUM6L57x4uG83SZpggBr/CEgzKri6qamxrAuu5AJg4/eaqPKInY7IYMHEzJ4MB1m3ETlnj2ULF1G8ZIlFH/3HY5PrTvIib8/gf36EZyaSvDAVIJSBxKQ1F3PkFKAdc+P2lbLkaqorrCSSEUJJVUlFFcUs7/KSjLuyaeksoT4EO/chkATBFinuCakQ71/8t/27qe4vEoruB6D/Dt0IGqiVY7cGEPV7t2UrllD2dq1lK5Zi+OTT9j3zjsA+IWHE5yaQtDAgQQPHEhwair2eL1viDoyAbYAYmyH15JpKZogqipg11oY8f8azMp0XkE9QM9gOqaJCP6dOuHfqRMRZ50FWAUGK7ZupXTNWkrXrqFszVryX/03VFndAvaEhAOtjIEDCR4wAL9QHcdS7YsmiP350HkodDuhwazMLAcBNj96d9SzWlRdYrMR2KsXgb16ETVxAgA1ZWWUrd9A2do1zsSxlqKvvrJW8PMjsGdPggamElw7ntGr1zFx/wvVfmmCiEiAP33ucVZmtoO+CeH46wC1aga/oCBCBg8iZPCBq/Gr9u2jLDOT0p/XULp2DcULF1lnSwESFERQ//4Ep6ZaiWPgQPy7dNGzplSboQmiEcYYMrMK+UNqgq9DUe2YPTqasJNPJuxk6y6Fxhgqs7IoW3OglbHvgw8wb74JWDWnXK2MtIEEpaZqTSnlM5ogGrFzXymO0kq9glq1KBEhoEsXArp0IeIPzntUV1ZSvnlznfGMvGUvQE0NAP5duxKcmkpgnz4E9EgmsEcPArp10+4p5XWaIBpRO0CdqldQKy8Tf3+C+vUjqF8/oi+cAkBNSQml69a5zprav3oVhQsWHFjJbiega1cCevSwEkaPHgT2SCagRw+9Ely1GE0QjcjMdmD3Ex2gVj7hFxpK6PDhhA4f7ppWXVxCxfbtVGzdQvnWrVRs3Ub51i0UL1kClZWu5ezx8VbC6NmDgOQerlaHvVMnHd9Qh0QTRCMyswrp1TGcIH+t+KjaBltYKMEpAwhOqVtx2FRVUblzpzNpbKV86zYqtmzBMW8+NUUH6gpJSAiBycl1kkdgj2QCundH9Opw5YEmCA+sAWoHp/ft4OtQlDoosdsJSEoiICkJTj/dNd0YQ3V+PuVbtlKxbauVQLZsZf/KFRTOm3dgAzabNS7iodWhZdGPbZogPNhVWEZ+SQWpXfSfQ7VfIoI9Lg57XByhI4bXmVezfz/l27ZRsXWbM3lYrY6SZcswbt1Vtrg4q9XR0znWkZyMf+cu+Ccm4BcU1NpvSbUyTRAeZGYVAjBAS2yoo5RfSIjrnt/uTHU1lVlZlG/ZYo1xbLNaHYWff0GNw1FnWVtMDP4JCfgnJuCfmIg9IQH/hET8ExPxT0zAFhOjYx7tnCYIDzKzHPgJ9EvQAWp1bBGbjYBu3Qjo1g1OO8013RhD9b59VGzdSmV2NpXZOVTm5FCZnU35tm0U/+87zP79dbcVGOhKIPYEK4lYCSQB/wRrmlbGbds0QXiwLttBz/gwQgL041EKnN1VMTHYYzwXjjPGUONwWEkjJ4fKrGzn82wriSxZSlVuboP1bPFxB1odCVbi8O+c6EogtqgobYX4kH4DerA2y8HInnG+DkOpdkNEsEVFWVeC9+vncZmaigqqdu9ukDyqsnMo37SJ4kWLMOXldbcbEnIgcbgnj04J2OPjsMfG4hcRoUnESzRB1LOnqIzdheUM0AvklGpRfgEB1sV9Xbt6nF/bjVWZnUNldhZVOTnO51ZCKduwger8/Abrib8/tthY7LGx2OJiscdaicMeF4stNg57XO28OGyRkXrvjkOgCaKeddnWAHVKopbYUKo1uXdj1b/Wo1ZNWRmVOTlU7dpFVV4+Vfl5VOfnH3iem0f5xk1U5ee7Sq/XYbdjj47GFudMIu5JJS7WSjTOebboaMR2bF8HpQminnWue0BoC0KptsYvKIjA5GQCk5ObXK52TKSqNnnk5dZNJHn5VOXnU75lC9V5eXVO7T2wMz9s0dF1WyP1Wia2mBhskVHYoiLxCw096rq6NEHUszbLQY+4UMIC9aNRqr1yHxMJ7NmzyWWNMdQUFVGVl091ft6BpOKWSKry86j47Xeq8vMxZWWeN2SzYYuIwBYRgV9UJLbISGwRzp+RkdgiI/Crneaab01rq2dz6bdgPZlZhQzuruWVlTpWiIjri50ezWiZlOx3JZLqvXupdhRS7XBYj0IHNQ4H1QUOqvfuo2LbdqoLC6kpLARjGo8hJMSKwZk0bFGRB5KJW4KxRUbi55Zg/MLCvNpq0QThZl9JBVkFpVx+Qndfh6KUaoNEBFtYKLawUAK6N/97wlRXU1NURHVhbTIppNpRQLXDQU1hoZVQXPMKqNj+myvp1D+zqw4/P2wREQQPHkzX559rgXdYlyYIN7UD1FriWynVksRmc3V5HaqasjKqHYXUFDoOtFRqWy3OFos9Pr7lg0YTRB1raweotcSGUqqN8AsKsupedWz94qF6QrCbzGwHXWOCiQzRO3UppZQmCDfrshykaOtBKaUATRAuhWWVbM/fT4qOPyilFKAJwmWds8S3JgillLJ4NUGIyNkisklENovIbR7mTxORXBFZ7Xxc6Tav2m36XG/GCVYFV4ABWmJDKaUAL57FJCI24DngTGAn8JOIzDXGrK+36AfGmOs8bKLUGJPurfjqy8xykBAZRFxYYGvtUiml2jRvtiCGA5uNMVuNMRXA+8B5XtzfEcnMLtTuJaWUcuPNBNEZ2OH2eqdzWn0TRWSNiMwWEfc6wEEiskJEfhCR8z3tQESuci6zItfDzUiaq6S8ii25xXoGk1JKufH1IPU8IMkYMxD4GnjTbV53Y8xQ4BLgKRFpUHHLGPOyMWaoMWZo/BFcSbghpxBjIKWzjj8opVQtbyaILMC9RdDFOc3FGJNvjKktNPIqMMRtXpbz51ZgMTDIW4FmOq+g1i4mpZQ6wJsJ4iegl4gki0gAcBFQ52wkEUlwezke2OCcHi0igc7nccBIoP7gdovJzC4kPjyQjhFB3tqFUkq1O147i8kYUyUi1wFfAjbgNWPMOhG5D1hhjJkL3CAi44EqYC8wzbl6P+AlEanBSmIPezj7qcVkZjn0DnJKKVWPV4v1GWMWAAvqTbvb7fntwO0e1vsOSPVmbLXKKqv5dU8xZ/bv2Bq7U0qpdsPXg9Q+V1RWxTmpCRzfI9bXoSilVJtyzJf7jg8P5JmLvTb+rZRS7dYx34JQSinlmSYIpZRSHmmCUEop5ZEmCKWUUh5pglBKKeWRJgillFIeaYJQSinlkSYIpZRSHokxxtcxtAgRyQV+O4JNxAF5LRROe6efRV36edSln8cBR8Nn0d0Y4/F+CUdNgjhSIrLCef+JY55+FnXp51GXfh4HHO2fhXYxKaWU8kgThFJKKY80QRzwsq8DaEP0s6hLP4+69PM44Kj+LHQMQimllEfaglBKKeWRJgillFIeHfMJQkTOFpFNIrJZRG7zdTy+JCJdRWSRiKwXkXUicqOvY/I1EbGJyCoRme/rWHxNRKJEZLaIbBSRDSJygq9j8iURmeH8P8kUkfdEJMjXMbW0YzpBiIgNeA4YC/QHLhaR/r6NyqeqgL8aY/oDxwN/OcY/D4AbgQ2+DqKNeBr4whjTF0jjGP5cRKQzcAMw1BiTAtiAi3wbVcs7phMEMBzYbIzZaoypAN4HzvNxTD5jjMkxxmQ4nxdhfQF09m1UviMiXYBzgFd9HYuviUgkcArwbwBjTIUxpsCnQfmeHQgWETsQAmT7OJ4Wd6wniM7ADrfXOzmGvxDdiUgSMAhY7uNQfOkp4G9AjY/jaAuSgVzgdWeX26siEurroHzFGJMFPAb8DuQADmPMV76NquUd6wlCeSAiYcBHwE3GmEJfx+MLInIusMcYs9LXsbQRdmAw8IIxZhBQAhyzY3YiEo3V25AMJAKhInKpb6Nqecd6gsgCurq97uKcdswSEX+s5PCOMeZjX8fjQyOB8SKyHavr8XQRedu3IfnUTmCnMaa2RTkbK2Ecq84Athljco0xlcDHwIk+jqnFHesJ4iegl4gki0gA1iDTXB/H5DMiIlh9zBuMMU/4Oh5fMsbcbozpYoxJwvq7WGiMOeqOEJvLGLML2CEifZyTRgPrfRiSr/0OHC8iIc7/m9EchYP2dl8H4EvGmCoRuQ74EusshNeMMet8HJYvjQQuA9aKyGrntDuMMQt8F5JqQ64H3nEeTG0FrvBxPD5jjFkuIrOBDKyz/1ZxFJbd0FIbSimlPDrWu5iUUko1QhOEUkopjzRBKKWU8kgThFJKKY80QSillPJIE4RSh0BEqkVktdujxa4mFpEkEclsqe0pdaSO6esglDoMpcaYdF8HoVRr0BaEUi1ARLaLyCMislZEfhSR45zTk0RkoYisEZFvRaSbc3pHEflERH52PmrLNNhE5BXnfQa+EpFgn70pdczTBKHUoQmu18V0ods8hzEmFXgWqxIswL+AN40xA4F3gGec058B/muMScOqaVR7BX8v4DljzACgAJjo1XejVBP0SmqlDoGIFBtjwjxM3w6cbozZ6ix4uMsYEysieUCCMabSOT3HGBMnIrlAF2NMuds2koCvjTG9nK9vBfyNMQ+0wltTqgFtQSjVckwjzw9FudvzanScUPmQJgilWs6Fbj+/dz7/jgO3opwKLHU+/xa4Blz3vY5srSCVai49OlHq0AS7VboF6x7Ntae6RovIGqxWwMXOaddj3YVtJtYd2WoroN4IvCwif8ZqKVyDdWcypdoMHYNQqgU4xyCGGmPyfB2LUi1Fu5iUUkp5pC0IpZRSHmkLQimllEeaIJRSSnmkCUIppZRHmiCUUkp5pAlCKaWUR/8f9oG6xutHq50AAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"model.plot(path=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"While the y-scale on the plot is a bit misleading when looking at the accuracy, there is no overfitting in there because the BCE cost is the same for both training and validation set at the end of the regression."
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"For code, maths and pictures behind the *RNN* layer, follow this link:\n",
"\n",
"* [Recurrent Neural Network (RNN)](https://epynn.net/RNN.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### LSTM-Dense"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now proceed with an *LSTM* layer composed of the 1 unit, all other things being equal."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"name = 'LSTM-1_Dense-2-softmax'\n",
"\n",
"se_hPars['learning_rate'] = 0.005\n",
"\n",
"lstm = LSTM(1)\n",
"\n",
"dense = Dense(2, softmax)\n",
"\n",
"layers = [embedding, lstm, dense]\n",
"\n",
"model = EpyNN(layers=layers, name=name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Initialize the model."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m--- EpyNN Check OK! --- \u001b[0m\r"
]
}
],
"source": [
"model.initialize(loss='BCE', seed=1, se_hPars=se_hPars.copy(), end='\\r')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Train for 50 epochs."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m\u001b[37mEpoch 9 - Batch 0/0 - Accuracy: 0.734 Cost: 0.57893 - TIME: 0.97s RATE: 1.03e+01e/s TTC: 0s \u001b[0m\n",
"\n",
"+-------+----------+----------+----------+-------+--------+-------+-----------------------------------+\n",
"| \u001b[1m\u001b[37mepoch\u001b[0m | \u001b[1m\u001b[37mlrate\u001b[0m | \u001b[1m\u001b[37mlrate\u001b[0m | \u001b[1m\u001b[32maccuracy\u001b[0m | | \u001b[1m\u001b[31mBCE\u001b[0m | | \u001b[37mExperiment\u001b[0m |\n",
"| | \u001b[37mLSTM\u001b[0m | \u001b[37mDense\u001b[0m | \u001b[1m\u001b[32mdtrain\u001b[0m | \u001b[1m\u001b[32mdval\u001b[0m | \u001b[1m\u001b[31mdtrain\u001b[0m | \u001b[1m\u001b[31mdval\u001b[0m | |\n",
"+-------+----------+----------+----------+-------+--------+-------+-----------------------------------+\n",
"| \u001b[1m\u001b[37m0\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.586\u001b[0m | \u001b[1m\u001b[31m0.563\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m1\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.580\u001b[0m | \u001b[1m\u001b[31m0.552\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m2\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.550\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m3\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m4\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m5\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m6\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m7\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m8\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m9\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014423_LSTM-1_Dense-2-softmax\u001b[0m |\n",
"+-------+----------+----------+----------+-------+--------+-------+-----------------------------------+\n"
]
}
],
"source": [
"model.train(epochs=10, init_logs=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The accuracy metrics are simply identical to what we have seen with a simple *RNN*, which is much faster to compute. It is not significantly better than what we obtained from a simple Perceptron, itself way faster to compute than the *RNN* based network."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuWUlEQVR4nO3deXwV9b3/8dcnCwRIkMiiRUBSFtliEMJSqQpSt9tKKypoXYq9Qm0vtqL1iq3X4lJ/XpdavXpVtIpesVKxWtxbJbjUNbhUQAUElIALImDCYrbP74+ZHE7CZIMcTkLeTx7nkTPf73xnPjMh53O+3znnO+buiIiI1JSS7ABERKR5UoIQEZFIShAiIhJJCUJERCIpQYiISCQlCBERiaQEISIikZQgRKRFMbOrzexLM/ss2bHs65Qgmikzm25mhWb2jZnNqVHX3sz+N/wj2WJmL8bVzTCzVWb2tZmtN7ObzCwtrn6NmW03s5Lw8fca255hZp+F7e8xs7ZxdVeZ2XtmVm5ms+qI/R4zczPrG1HXz8x2mNkDcWXfN7OXzWxzuO+7zSwrrv46M1sbxvSxmf2mAeevdxhDWkRdpzDGz8ys2MyWm9lMM+sVd15KwvZb45aPMLM5YfkPa2zzprB8Sh0xNej81Wgzxcwq4mJYbWb3mln/hrTfW8ysm5n9Ofw/t8XM/mlmoxKwn17ARcAgdz8wPD8vN/V+JKAE0XytB64G7omomw3sDwwMf86Iq1sADHP3jsAQIA/4ZY32J7p7Zvg4tqrQzI4DZgLjgYOBbwNXxLVbCfwn8GRtQZvZd4E+dRzXbcCbNcr2IzjW7uExHQRcH1f/J2BAeEyHA2eY2cQ69lGfm4DMcF/7AROAle7+Sdx5yQzXzYsreyksWw6cXbWxMAlNAj6qZ7/1nr9avBrGsx/wPWA7sNjMhjRyO4mUSfB7HU7wf/I+4Ekzy6yzVeP1Aja6+xdNvF2JoATRTLn7X939MWBjfLmZDSB4QZvm7hvcvcLdF8e1+8jdN1etDlQCu7yTr8VPgD+5+1J33wRcBUyJ2/Z97v40UBzVOHyh/B/g/FrqTwM2A8/XONYH3f0Zd98W7vcuYExc/YfuvjWuSWOOKcoI4EF33+Tule7+gbvPb0T7x4Hvmll2uHw88C+gziGP+s5ffcLf9Ufu/gvgBWBWVZ2ZjTazV8Je2LtmNjaublHYe/ln2GP6u5l1CesyzOwBM9sYtn3TzA4I6/Yzsz+Z2admts6CoZ3UWmJb5e5/cPdPwzhnA22AQ6LWt8BNZvZF2DN8ryrhhfu938w2hD3Gy8wsxcy+B/wD6B72puYBdwDfCZc3h+3nWNDDfjos/6eZHWhmfzSzTWb2gZkdFhfLTDP7KDw3y8zspLi6283skbjl/zaz583MGvO7a6mUIFqekcDHwBUWDDG9Z2Ynx69gZj82s6+BLwl6EHfW2Mbc8I/v72aWF1c+GHg3bvld4AAz69zA2GYAL7r7v2pWmFlH4ErgwgZs50hgaY32M82sBCgCOgAPNjCmKK8Bvzezc8ys32603wH8DTgtXD4buH8P4tkdfwWOADCzgwh6JVcTvHv/NfCImXWNW//HwDlAN4IX7l+H5T8h6Jn0BDoD5xH0UADmAOUEyfgw4Fjg3IYEZ2ZDw/2srGWVYwl+z/3D/U9i55uh/wnLvg0cRXB+z3H354ATgPVhj25yGO+r4XKnuO1PAi4DugDfAK8Cb4XL84E/xK37EcG53I+gx/yAmX0rrLsIyA2Hso4A/h34ibeSSeyUIFqeHgRDR1sIhmSmA/eZ2cCqFcJ35B0J/vjuAD6Pa38G0JtgCKkAeNbMOoV1meF2q1Q9z6IeZtYT+BlweS2rXEXQOymqZzvHELxoVduOu18bxjEM+L8acTbW+cBcgnO3zMxWmtkJjdzG/cDZ4bk7CnhsD+LZHesJkgHAmcBT7v5U2CP6B1AI/Fvc+ve6+3J33w78BRgalpcRJIa+Vb1Rd/867EX8G3CBu28Nh3RuYmdSrFX4ZuD/gCvcvbbfUxnB73MAYO7+vrt/GvZQTgMudfdid18D3Aic1aCzstOj4bHsAB4Fdrj7/e5eAcwjSHgAuPvD7r4+PHfzgBUEb8Rw923hvv8APACcX9//4X2JEkTLs53gj+tqdy919xcIXuiPrbmiu68geCf+v3Fl/3T37eFwzv8jGPI5IqwuATrGbaLqeUOGRP4IXBn1ghC+m/wewQtMrcxsNEHP4BR3Xx5xPO7ubxOcgytq1jdUePzXuPtwghfHvwAPm9n+9TSN38bLQFfgt8AT4Qtv/LEsjbuwfETkRvbMQcBX4fODgVPDIaLN4VDLd4Fvxa0fP/y1jeDNAAQv5M8CD4UXmK8zs/Rwm+nAp3HbvJOgB1Lr8ZlZO4IhuNfC/19Ere/uC4FbCa5JfWFms8PE0iXc78dx8X4cHm9jxL8p2h6xHLs2YmZnm9k7ccc5JIwDAHd/HVhFMGT7l0bG0aLt8gkPafZ2Gb4B6uruplH3RWMn+I8PQTLJY+cfQR7wubtvjGpYw3iCcfnr4speNbNfEbyo9AY+CYduM4FUMxvk7sMAwjHhBcBP3b3aNYrdOKYGC98tXwNcCuSw80W3IR4g6OmMi9ju4KaIrw4nAVUXzdcC/+fuUxu7EXcvI0i2V5hZb+Ap4MPw5zdAF3cvj2i3y/FZ8Im3xwiGAX9W3/rufgtwi5l1I/g/dzHBdZUyggS1LFy1F7CutkOo5xDrZGYHE1zzGk8wVFVhZu+w828CM/sPoC1Br+0/gf8Xsal9knoQzZSZpZlZBpBK8GKaYcFF4BeBT4BLw3XGELxAPRu2Ozf8g8PMBhG88D0fLvcyszFm1ibc3sUE75T+Ge72fuDfzWxQOHRyGcE4dFVM6WFMKUBauI2qi5b9CRLKUHYOX5xI0L2fTfCCXlV3B8GY+XHhdocAzxB03x+vcR5SzOxnZpZtgZHAf1DjQncd2oZxVj1SzOy/zGxE1XkAfkXQk/qwgduscgtwDMHvpF71nL+GtE81sxwz+x9gLDt7UQ8AJ5rZceE6GWY21sx6NGCb48wsN4zja4IX50p3/xT4O3CjmXUMz1sfMzuqtmMjGNvfTjBGX1nPfkeY2aiw3VaC6zqV4RDQXwiuEWWFL+AXhscY5XOgh5m1qe9Ya9GBIMlsCOM6h6AHURVnf4JrO2cSDDX9Z9gjbh3cXY9m+CB4J+U1HrPCusEEF922ErzLOimu3b0EfzRbgTUEHxfNiGv3r7BuI8GLbH6N/V4Ytv863FbbuLo5ETFNqSV+JxjXru3YHqgRcyXBEFfVY2lYl0KQPL4Ky5cDvyEYt67r/PWOiNUJhrouA5aEx/gVsAg4vCHHEJ6Dq2vZ58u1nY/Gnr+4NlOAivDYtxIMt9wHDKyx3iiCTzZ9RfBi9yTQK6xbBJxbY5svh89PJ0iMW8Pf+y1AWli3H3A7QY9gC/A2cFotcR4VHs+2Gr/HI2pZf3z4f7GE4MMUc4HMsC6bICFsIOgdXQ6khHVjgaK47bQJj/Ur4Muo3xHBhfVFcct9gfK45d9XtSe41vBC2CYNeAOYGbfuz4H3iPu72JcfFh60iIhINRpiEhGRSEoQ0mKZ2RlWfWqMqsfS+ls3H2Z2Ry3HcUeyY5PWTUNMIiISaZ/5mGuXLl28d+/eyQ5DRKRFWbx48Zfu3jWqbp9JEL1796awsDDZYYiItChm9nFtdboGISIikZQgREQkkhKEiIhEUoIQEZFIShAiIhJJCUJERCIpQYiISKR95nsQe+TpmfDZe8mOQkRk9xyYCydc2+SbVQ9CREQiqQcBCcm8IiItnXoQIiISSQlCREQiKUGIiEgkJQgREYmkBCEiIpGUIEREJJIShIiIREpogjCz483sQzNbaWYzI+pvMrN3wsdyM9scV1cRV7cgkXGKiMiuEvZFOTNLBW4DjgGKgDfNbIG7L6tax91nxK1/PnBY3Ca2u/vQRMUnIiJ1S2QPYiSw0t1XuXsp8BDwwzrWPx34cwLjERGRRkhkgjgIWBu3XBSW7cLMDgZygIVxxRlmVmhmr5nZj2ppNy1cp3DDhg1NFLaIiEDzuUh9GjDf3Sviyg5293zgx8AfzaxPzUbuPtvd8909v2vXrnsrVhGRViGRCWId0DNuuUdYFuU0agwvufu68OcqYBHVr0+IiEiCJTJBvAn0M7McM2tDkAR2+TSSmQ0AsoFX48qyzaxt+LwLMAZYVrOtiIgkTsI+xeTu5WY2HXgWSAXucfelZnYlUOjuVcniNOAhd/e45gOBO82skiCJXRv/6ScREUk8q/663HLl5+d7YWFhssMQEWlRzGxxeL13F83lIrWIiDQzShAiIhJJCUJERCIpQYiISCQlCBERiaQEISIikZQgREQkUsK+KNeSXPH4Upat/zrZYYiI7JZB3TvyuxMHN/l21YMQEZFI6kFAQjKviEhLpx6EiIhEUoIQEZFIShAiIhJJCUJERCIpQYiISCQlCBERiaQEISIikZQgREQkkhKEiIhEUoIQEZFIShAiIhJJCUJERCIpQYiISCQlCBERiaQEISIikZQgREQkkhKEiIhEUoIQEZFIShAiIhJJCUJERCIpQYiISCQlCBERiaQEISIikZQgREQkkhKEiIhEUoIQEZFIShAiIhIpoQnCzI43sw/NbKWZzYyov8nM3gkfy81sc1zdT8xsRfj4SSLjFBGRXaUlasNmlgrcBhwDFAFvmtkCd19WtY67z4hb/3zgsPD5/sDvgHzAgcVh202JildERKpLZA9iJLDS3Ve5eynwEPDDOtY/Hfhz+Pw44B/u/lWYFP4BHJ/AWEVEpIZEJoiDgLVxy0Vh2S7M7GAgB1jYmLZmNs3MCs2scMOGDU0StIiIBJrLRerTgPnuXtGYRu4+293z3T2/a9euCQpNRKR1SmSCWAf0jFvuEZZFOY2dw0uNbSsiIgmQyATxJtDPzHLMrA1BElhQcyUzGwBkA6/GFT8LHGtm2WaWDRwblomIyF6SsE8xuXu5mU0neGFPBe5x96VmdiVQ6O5VyeI04CF397i2X5nZVQRJBuBKd/8qUbGKiMiuLO51uUXLz8/3wsLCZIchItKimNlid8+PqmsuF6lFRKSZUYIQEZFIShAiIhJJCUJERCIl7FNMIrLvKCsro6ioiB07diQ7FNlNGRkZ9OjRg/T09Aa3UYIQkXoVFRWRlZVF7969MbNkhyON5O5s3LiRoqIicnJyGtxOQ0wiUq8dO3bQuXNnJYcWyszo3Llzo3uAShAi0iBKDi3b7vz+lCBEpMWZNWsWN9xwAwBz5sxh/fr1jd7GHXfcwf3339/Uoe1TdA1CRFq0OXPmMGTIELp3775LXUVFBampqZHtzjvvvESHttvKy8tJS0v+y7N6ECLSIvz+97+nf//+fPe73+XDDz8EYP78+RQWFnLGGWcwdOhQtm/fTu/evbnkkksYNmwYDz/8MHfddRcjRowgLy+Pk08+mW3btgHVeyFjx47lkksuYeTIkfTv35+XXnppl/2XlJQwfvx4hg0bRm5uLn/7299idffffz+HHnooeXl5nHXWWQB8/vnnnHTSSeTl5ZGXl8crr7zCmjVrGDJkSKzdDTfcwKxZs2IxXHDBBeTn53PzzTfz+OOPM2rUKA477DC+973v8fnnn8fiOOecc8jNzeXQQw/lkUce4Z577uGCCy6Ibfeuu+5ixozYDTt3W/JTlIi0KFc8vpRl679u0m0O6t6R3504uNb6xYsX89BDD/HOO+9QXl7OsGHDGD58OKeccgq33norN9xwA/n5O6cT6ty5M2+99RYAGzduZOrUqQBcdtll/OlPf+L888/fZR/l5eW88cYbPPXUU1xxxRU899xz1eozMjJ49NFH6dixI19++SWjR49mwoQJLFu2jKuvvppXXnmFLl268NVXwbyiv/zlLznqqKN49NFHqaiooKSkhE2b6r5rcmlpKVVzym3atInXXnsNM+Puu+/muuuu48Ybb+Sqq65iv/3247333outl56ezu9//3uuv/560tPTuffee7nzzjvrO+31UoIQkWbvpZde4qSTTqJ9+/YATJgwoc71J0+eHHu+ZMkSLrvsMjZv3kxJSQnHHXdcZJuJEycCMHz4cNasWbNLvbvzm9/8hhdffJGUlBTWrVvH559/zsKFCzn11FPp0qULAPvvvz8ACxcujF3jSE1NZb/99qs3QcTHXVRUxOTJk/n0008pLS2NfTz1ueee46GHHoqtl52dDcDRRx/NE088wcCBAykrKyM3N7fOfTWEEoSINEpd7/Sbiw4dOsSeT5kyhccee4y8vDzmzJnDokWLItu0bdsWCF7My8vLd6mfO3cuGzZsYPHixaSnp9O7d+9Gf2w0LS2NysrK2HLN9vFxn3/++Vx44YVMmDCBRYsWxYaianPuuedyzTXXMGDAAM4555xGxVUbXYMQkWbvyCOP5LHHHmP79u0UFxfz+OOPx+qysrIoLi6utW1xcTHf+ta3KCsrY+7cubsdw5YtW+jWrRvp6ekUFBTw8ccfA8E794cffpiNGzcCxIaYxo8fz+233w4EF8u3bNnCAQccwBdffMHGjRv55ptveOKJJ+rc30EHHQTAfffdFys/5phjuO2222LLVb2SUaNGsXbtWh588EFOP/303T7OeEoQItLsDRs2jMmTJ5OXl8cJJ5zAiBEjYnVTpkzhvPPOi12krumqq65i1KhRjBkzhgEDBux2DGeccQaFhYXk5uZy//33x7Y1ePBgfvvb33LUUUeRl5fHhRdeCMDNN99MQUEBubm5DB8+nGXLlpGens7ll1/OyJEjOeaYY+qMZ9asWZx66qkMHz48NnwFwXWUTZs2MWTIEPLy8igoKIjVTZo0iTFjxsSGnfaUbhgkIvV6//33GThwYLLDkHr84Ac/YMaMGYwfPz6yPur3qBsGiYjswzZv3kz//v1p165drclhd+gitYhIC9epUyeWL1/e5NtVD0JERCIpQYiISCQlCBERiaQEISIikZQgRKTFiZ9or7GmTJnC/PnzmziifVODE4SZtU9kICIirUHUNB7NVb0JwswON7NlwAfhcp6Z/W/CIxMRiRM13fcHH3zAyJEjY+usWbMmNkndlVdeyYgRIxgyZAjTpk2jvi8F1zYteNS03RA9xXfN3klmZiYAixYt4ogjjmDChAkMGjQIgB/96EcMHz6cwYMHM3v27FibZ555hmHDhpGXl8f48eOprKykX79+bNiwAYDKykr69u0bW06khnwP4ibgOGABgLu/a2ZHJjQqEWm+np4Jn73XtNs8MBdOuLbW6tqm+x4wYAClpaWsXr2anJwc5s2bF5sRdfr06Vx++eUAnHXWWTzxxBOceOKJte5j4sSJkdOCR03bvXTp0sgpvuvy1ltvsWTJktisrPfccw/7778/27dvZ8SIEZx88slUVlYydepUXnzxRXJycvjqq69ISUnhzDPPZO7cuVxwwQU899xz5OXl0bVr1waf3t3VoCEmd19bo6giAbGIiESKn+67Y8eO1ab7njRpEvPmzQOoliAKCgoYNWoUubm5LFy4kKVLl9a5jyVLlnDEEUeQm5vL3LlzY+svXLiQn//858DOabtrm+K7LiNHjowlB4BbbrmFvLw8Ro8ezdq1a1mxYgWvvfYaRx55ZGy9qu3+9Kc/jU0dfs899zTZbK31aUgPYq2ZHQ64maUDvwLeT2xYItJs1fFOPxkmT57MqaeeysSJEzEz+vXrx44dO/jFL35BYWEhPXv2ZNasWfVOzd3QacHrEj+dd2VlJaWlpbG6+Km8Fy1axHPPPcerr75K+/btGTt2bJ3x9ezZkwMOOICFCxfyxhtv7NGstI3RkB7EecB/AAcB64Ch4bKIyF5R13Tfffr0ITU1lauuuirWe6h6se3SpQslJSUN+tRSbdOCR03bXdsU371792bx4sUALFiwgLKyssh9bdmyhezsbNq3b88HH3zAa6+9BsDo0aN58cUXWb16dbXtQnC/hzPPPJNTTz211vtsN7V6E4S7f+nuZ7j7Ae7ezd3PdPeNeyM4ERGoe7pvCHoRDzzwAJMmTQKCuYmmTp3KkCFDOO6443ZZP0pt04JHTdtd2xTfU6dO5YUXXiAvL49XX321Wq8h3vHHH095eTkDBw5k5syZjB49GoCuXbsye/ZsJk6cSF5eXrU7zE2YMCF2P+q9pd7pvs3sXmCXldz9p4kKandoum+RxNF038lXWFjIjBkzeOmll3Z7G42d7rsh1yDib3mUAZwErN/tCEVEpFGuvfZabr/99r127aFKvQnC3R+JXzazPwMvJywiERGpZubMmcycOXOv73d3ptroB3Rr6kBERKR5qbcHYWbFBNcgLPz5GXBJguMSEZEka8gQU9beCERERJqXWhOEmQ2rq6G7v1Xfxs3seOBmIBW42913+YaNmU0CZhH0Tt519x+H5RVA1ff5P3H3CTXbiohI4tTVg7ixjjoHjq5rw2aWCtwGHAMUAW+a2QJ3Xxa3Tj/gUmCMu28ys/hrG9vdfWg98YtIKzRr1iwyMzP59a9/zZw5czj22GPp3r17o7Zxxx130L59e84+++wGrb9mzRoGDhzIIYccgrvToUMH7r33Xg455BAAnn76af7rv/6Lbdu20bZtW44++mhuvPFGZs2axV133VVt7qRFixbRqVOnRsWbDLUmCHcft4fbHgmsdPdVAGb2EPBDYFncOlOB29x9U7jPL/ZwnyLSysyZM4chQ4ZEJoiKiopav3V83nnnNXpfffr04Z133gHgzjvv5JprruG+++5jyZIlTJ8+nSeffJIBAwZQUVFRbYbWGTNm8Otf/7rR+0u2Bn2KycyGmNkkMzu76tGAZgcB8ZP8FYVl8foD/c3sn2b2WjgkVSXDzArD8h/VEte0cJ3CvTH1rYgkT9R03/Pnz6ewsJAzzjiDoUOHsn37dnr37s0ll1zCsGHDePjhh2udxjv+pkNjx47lkksuYeTIkfTv379BX0b7+uuvyc7OBuC6667jt7/9bewb2KmpqbEJ/lqyhnyK6XfAWGAQ8BRwAsH3IO5vov33C7ffA3jRzHLdfTNwsLuvM7NvAwvN7D13/yi+sbvPBmZD8E3qJohHROrx32/8Nx989UGTbnPA/gO4ZGTtH46sbbrvU045hVtvvZUbbriB/PydXwbu3Lkzb70VXCbduHFj5DTeNZWXl/PGG2/w1FNPccUVV/Dcc8/tss5HH33E0KFDKS4uZtu2bbz++utAMBPsRRddVGv8N910Ew888AAA2dnZFBQUNOCsJF9Dvkl9CpAHvO3u55jZAcADDWi3DugZt9wjLItXBLzu7mXAajNbTpAw3nT3dQDuvsrMFgGHAR8hIq1O/HTfQLXpvqPEz2G0ZMkSLrvsMjZv3kxJSQnHHXdcZJuJEycCMHz4cNasWRO5TvwQ07x585g2bRrPPPNMvfG31CGmhiSIHe5eaWblZtYR+ILqL/y1eRPoZ2Y5BInhNODHNdZ5DDgduNfMuhAMOa0ys2xgm7t/E5aPAa5r0BGJSELV9U6/uYifJK+h03i3bdsWCIaHGnJb0AkTJsQmzhs8eDCLFy8mLy9vz4NvRmq9BmFmt5nZd4E3zKwTcBewGHgLeLW+Dbt7OTAdeJbg/hF/cfelZnalmVWl/2eBjeEtTQuAi8OZYgcChWb2blh+bfynn0Skdalruu+srCyKi4trbVvbNN576uWXX6ZPnz4AXHzxxVxzzTUsX74cCO4FcccddzTZvpKlrh7EcuB6oDuwFfgzwUdWO7r7vxqycXd/iuC6RXzZ5XHPHbgwfMSv8wqQ25B9iMi+L366727dulWbvnvKlCmcd955tGvXjldf3fW9a9U03l27dmXUqFF1JpP6VF2DcHfatGnD3XffDcChhx7KH//4R04//XS2bduGmfGDH/wg1i7+GgTAY489Ru/evXc7jr2lIdN9H0wwPHQa0I4gUTzo7isSH17DabpvkcTRdN/7hsZO992QGwZ97O7/7e6HEVwv+BHQtB9hEBGRZqfeBGFmaWZ2opnNBZ4GPgQmJjwyERFJqrrmYjqGoMfwb8AbwEPANHffupdiExGRJKrrIvWlwIPARVVTYYiISOtR11xMdU7GJyIi+7bduaOciIi0AkoQItLixE+011hTpkxh/vz5keU5OTkMHTqUAQMGcMUVV8TqSkpK+NnPfkafPn0YPnw4Y8eOjc3DlJqaytChQ2OPa6/d5bY3LVZDptoQEWkVrr/+ek455RR27NjBoEGDOPvss8nJyeHcc88lJyeHFStWkJKSwurVq1m2LJjcoV27drH5mfY16kGISIsQNd33Bx98wMiRI2PrrFmzhtzcYBKGK6+8khEjRjBkyBCmTZtGfV8Kjrdjxw4gmNPpo48+4vXXX+fqq68mJSV4yczJyeH73/9+Ux1as6UehIg0ymfXXMM37zftd2XbDhzAgb/5Ta31tU33PWDAAEpLS1m9ejU5OTnMmzcvNpPr9OnTufzyYGafs846iyeeeIITTzyxzjguvvhirr76alauXMkvf/lLunXrxmuvvcbQoUNrvfHQ9u3bGTp0aGz50ksvrTabbEumHoSINHvx03137Nix2nTfkyZNYt68eQDVEkRBQQGjRo0iNzeXhQsXsnTp0nr3c/311/POO+/w2Wef8fzzz/PKK6/U26ZqiKnqsa8kB1APQkQaqa53+skwefJkTj31VCZOnIiZ0a9fP3bs2MEvfvELCgsL6dmzJ7NmzYoNGzVEZmYmY8eO5eWXX+bkk0/m3XffrfP2pfsq9SBEpNmra7rvPn36kJqaylVXXRV7916VDLp06UJJSUnkp5bqUl5ezuuvv06fPn3o06cP+fn5/O53v4tdx1izZg1PPvlkEx1d86UehIg0e3VN9w1BL+Liiy9m9erVAHTq1ImpU6cyZMgQDjzwwF3Wr03VNYjS0lLGjx8fu8vc3XffzUUXXUTfvn1p164dXbp04frrrwd2vQZx/PHH7zMfda13uu+WQtN9iySOpvveNzT5dN8iItI6KUGIiEgkJQgREYmkBCEiDbKvXK9srXbn96cEISL1ysjIYOPGjUoSLZS7s3HjRjIyMhrVTh9zFZF69ejRg6KiIjZs2JDsUGQ3ZWRk0KNHj0a1UYIQkXqlp6eTk5OT7DBkL9MQk4iIRFKCEBGRSEoQIiISSQlCREQiKUGIiEgkJQgREYmkBCEiIpGUIEREJJIShIiIRFKCEBGRSEoQIiISSQlCREQiKUGIiEgkJQgREYmU0ARhZseb2YdmttLMZtayziQzW2ZmS83swbjyn5jZivDxk0TGKSIiu0rY/SDMLBW4DTgGKALeNLMF7r4sbp1+wKXAGHffZGbdwvL9gd8B+YADi8O2mxIVr4iIVJfIHsRIYKW7r3L3UuAh4Ic11pkK3Fb1wu/uX4TlxwH/cPevwrp/AMcnMFYREakhkQniIGBt3HJRWBavP9DfzP5pZq+Z2fGNaCsiIgmU7FuOpgH9gLFAD+BFM8ttaGMzmwZMA+jVq1ci4hMRabUS2YNYB/SMW+4RlsUrAha4e5m7rwaWEySMhrTF3We7e76753ft2rVJgxcRae0SmSDeBPqZWY6ZtQFOAxbUWOcxgt4DZtaFYMhpFfAscKyZZZtZNnBsWCYiIntJwoaY3L3czKYTvLCnAve4+1IzuxIodPcF7EwEy4AK4GJ33whgZlcRJBmAK939q0TFKiIiuzJ3T3YMTSI/P98LCwuTHYaISItiZovdPT+qTt+kFhGRSEoQIiISSQlCREQiKUGIiEgkJQgREYmkBCEiIpGUIICi4iIqvTLZYYiINCvJnosp6Tbv2MwJfz2Bdmnt+PZ+36Zvp770y+5H30596dupL93ad8PMkh2miMhe1+oTRHpqOlcefiUrNq9g5aaVvLL+Ff720d9i9VltsujXKUwY2X1jiSM7IzuJUYuIJF6rTxAd0jtwUr+TqpVt3rGZlZtXxh4rNq3g6TVPU7y8OLZO54zO9M3uu0vy6JDeYW8fgohIQrT6BBGlU0Yn8g/MJ//And8+d3c2bN/Ayk0rg97G5pWs3LSSR1Y8wvby7bH1unfoXq2n0S+7Hzn75dA2tW0yDkVEZLcpQTSQmdGtfTe6te/G4QcdHiuv9ErWlaxj5aawtxEmj1fWv0J5ZTkAKZZCr6xe9MvuR59OfYLE0akfvTr2Ii1FvwIRaZ706rSHUiyFnlk96ZnVk3G9xsXKyyrLWPv12mq9jRWbVvD8J8/HPjGVnpJOzn45O3saHXNol9aO1JRU0lLSSLVU0lPSY8/TUtKqPaqVWfBTF9RFpKloNte9bEf5DlZvWb2ztxH2PD7d+mmTbD/VUmOJIzUlTDCWVi3ppKWkkZ6SXi3BVNWnWbBOVaIxbJekE19mxP00qpeZVa8Py+K3E18Wtd2msKdJsylj2RPNJY7mQm+Gduqe2Z2fDvnpbrWtazZX9SD2soy0DAZ2HsjAzgOrlReXFvNJ8SeUVpRSXllOeWU5FV4Re17uYVnlrmX1rVvhFZRVllVbr6KygjIviz3/pvwbtlZuDeq9nJpvHKqWPfwXX1ZVvst6cc+jtlOzXWy92I89f/Oyp9toLm+gmuJcyL5rYOeBu50g6qIE0UxktclicOfByQ5DRCRG36QWEZFIShAiIhJJCUJERCIpQYiISCQlCBERiaQEISIikZQgREQkkhKEiIhEUoIQEZFIShAiIhJJCUJERCIpQdB8JmQTEWlOWv1kfZU7drDq+z+g/ciRZI4bS+aYMaR00G1DRURafYKo+Ppr2g0dSvHzz7Pl0Uex9HTajxpF5rixZI0bR3r37skOUUQkKXTDoJCXlbHtrbcpKSigpKCA0o8/BqDtIYfEkkVGbi6WolE5Edl31HXDICWIWnyzanUsWWx7+22oqCC1SxcyjzqSrHHj6HD44aS0b99k+xMRSQYliD1UsXkzJS+9FCSMl16msrgYa9OG9qNHkTVuHJljx5L+rW8lZN8iIomkBNGEvKyMbYsXU1JQQHHBIso++QSAtgMHkjVuLJljx5IxZIiGokSkRVCCSBB3p3TVKkoWLaK4oIDtb70NlZWkdu1C5lFHBUNR3/mOhqJEpNlSgthLyjdtYutLL1FcUMDWl16msqQEa9u2+lDUgQcmNUYRkXhKEEngpaVsW7yY4oICSgoWUbZ2LQBtBw0ka+w4MseNI2PwIA1FiUhSKUEkmbtT+tFHsWSx/Z13oLKStK5dyRw7lsxx4+jwndGktGuX7FBFpJVJWoIws+OBm4FU4G53v7ZG/RTgemBdWHSru98d1lUA74Xln7j7hLr21ZwTRE3lmzZR8sILlBQsYuvLL1O5dSuWkUH7ESNI69aV1KyOpGRlkpqVRUpm1s7nWVk7f2ZmYunpyT4UEWnhkpIgzCwVWA4cAxQBbwKnu/uyuHWmAPnuPj2ifYm7ZzZ0fy0pQcTz0lK2vvkmJQWL2PbGG1Rs3kxFSQm+bVu9bS0jI0weYULJrEoimXGJpXqySc3K3JloMjOxtFb/ZXqRVq2uBJHIV4eRwEp3XxUG8RDwQ2BZna1aGWvThswxY8gcM6ZauZeXU1lSQkVxMZXFxVQUl1BZUhwul1BR/DWVsbKSYJ2SYsrWr6eiJFjHd+yof//t25OamRnrlaRkZZHSoQOWlhYkj7RULDUNS0uFtLSdz1NTsbT0nc9Ta65ftV7w09LSwjZpWGq4rarnqWlYetXz1J37TkkFAzODlBQwA7NgOa4stlxHmZkl6lcoss9KZII4CFgbt1wEjIpY72QzO5KgtzHD3avaZJhZIVAOXOvuj9VsaGbTgGkAvXr1asLQk8/S0kjt1InUTp12exteWkrF1q1B8vi6uFqCiT3/ujiWUCpLiqnYsoWy9evxinIor8ArKvDycigvD55XVEBZWfCzsrLpDnhvqJk0qsoakmjitxH5s+pHbfU1fu5OmzpynNVZuRt1u9OmNWom13AzBhzCQX/4Q5NvN9njC48Df3b3b8zsZ8B9wNFh3cHuvs7Mvg0sNLP33P2j+MbuPhuYDcEQ094MvCWwNm1Ia9MGsrMTsn2vrIQwgXhFxc4kEp9Qyivw8rJwvQqoKA/WL68IklBV+/L451XrVQAO7sG+nOAP0iuDMneo9FiZuwfrVFYCVW2iyhqxHd+ZBGPDsbGf1FiOFdS9flSbWtvWWD/yF7GbdUTX1Tns3Fz+ytybT6JqBmGk90zMG+REJoh1QM+45R7svBgNgLtvjFu8G7gurm5d+HOVmS0CDgOqJQhJLktJCd6J62K5yD4pkR/CfxPoZ2Y5ZtYGOA1YEL+CmcVPYDQBeD8szzaztuHzLsAYdO1CRGSvSlgPwt3LzWw68CzBx1zvcfelZnYlUOjuC4BfmtkEgusMXwFTwuYDgTvNrJIgiV0b/+knERFJPH1RTkSkFavrY66a50FERCIpQYiISCQlCBERiaQEISIikZQgREQk0j7zKSYz2wB8vAeb6AJ82UThtHQ6F9XpfFSn87HTvnAuDnb3rlEV+0yC2FNmVljbR71aG52L6nQ+qtP52GlfPxcaYhIRkUhKECIiEkkJYqfZyQ6gGdG5qE7nozqdj5326XOhaxAiIhJJPQgREYmkBCEiIpFafYIws+PN7EMzW2lmM5MdTzKZWU8zKzCzZWa21Mx+leyYks3MUs3sbTN7ItmxJJuZdTKz+Wb2gZm9b2bfSXZMyWRmM8K/kyVm9mczy0h2TE2tVScIM0sFbgNOAAYBp5vZoORGlVTlwEXuPggYDfxHKz8fAL8ivJGVcDPwjLsPAPJoxefFzA4Cfgnku/sQgnvenJbcqJpeq04QwEhgpbuvcvdS4CHgh0mOKWnc/VN3fyt8XkzwAnBQcqNKHjPrAXyf4Ha4rZqZ7QccCfwJwN1L3X1zUoNKvjSgnZmlAe2B9UmOp8m19gRxELA2brmIVvyCGM/MehPcB/z1JIeSTH8E/hOoTHIczUEOsAG4Nxxyu9vMOiQ7qGRx93XADcAnwKfAFnf/e3KjanqtPUFIBDPLBB4BLnD3r5MdTzKY2Q+AL9x9cbJjaSbSgGHA7e5+GLAVaLXX7Mwsm2C0IQfoDnQwszOTG1XTa+0JYh3QM265R1jWaplZOkFymOvuf012PEk0BphgZmsIhh6PNrMHkhtSUhUBRe5e1aOcT5AwWqvvAavdfYO7lwF/BQ5PckxNrrUniDeBfmaWY2ZtCC4yLUhyTEljZkYwxvy+u/8h2fEkk7tf6u493L03wf+Lhe6+z71DbCh3/wxYa2aHhEXjgWVJDCnZPgFGm1n78O9mPPvgRfu0ZAeQTO5ebmbTgWcJPoVwj7svTXJYyTQGOAt4z8zeCct+4+5PJS8kaUbOB+aGb6ZWAeckOZ6kcffXzWw+8BbBp//eZh+cdkNTbYiISKTWPsQkIiK1UIIQEZFIShAiIhJJCUJERCIpQYiISCQlCJFGMLMKM3sn7tFk3yY2s95mtqSptieyp1r19yBEdsN2dx+a7CBE9gb1IESagJmtMbPrzOw9M3vDzPqG5b3NbKGZ/cvMnjezXmH5AWb2qJm9Gz6qpmlINbO7wvsM/N3M2iXtoKTVU4IQaZx2NYaYJsfVbXH3XOBWgplgAf4HuM/dDwXmAreE5bcAL7h7HsGcRlXf4O8H3Obug4HNwMkJPRqROuib1CKNYGYl7p4ZUb4GONrdV4UTHn7m7p3N7EvgW+5eFpZ/6u5dzGwD0MPdv4nbRm/gH+7eL1y+BEh396v3wqGJ7EI9CJGm47U8b4xv4p5XoOuEkkRKECJNZ3Lcz1fD56+w81aUZwAvhc+fB34Osfte77e3ghRpKL07EWmcdnEz3UJwj+aqj7pmm9m/CHoBp4dl5xPche1igjuyVc2A+itgtpn9O0FP4ecEdyYTaTZ0DUKkCYTXIPLd/ctkxyLSVDTEJCIikdSDEBGRSOpBiIhIJCUIERGJpAQhIiKRlCBERCSSEoSIiET6/55iUaQPbNpnAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"model.plot(path=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"By contrast with the RNN-based network, we observe here a slight overfitting because the cost is lower for the training dataset compared to the validation dataset.\n",
"\n",
"For code, maths and pictures behind the *LSTM* layer, follow this link:\n",
"\n",
"* [Long Short-Term Memory (LSTM)](https://epynn.net/LSTM.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### GRU-Dense"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now proceed with a GRU layer, all other things being equal."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"name = 'GRU-1_Dense-2-softmax'\n",
"\n",
"se_hPars['learning_rate'] = 0.005\n",
"\n",
"gru = GRU(1)\n",
"\n",
"flatten = Flatten()\n",
"\n",
"dense = Dense(2, softmax)\n",
"\n",
"layers = [embedding, gru, dense]\n",
"\n",
"model = EpyNN(layers=layers, name=name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Initialize the network."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m--- EpyNN Check OK! --- \u001b[0m\r"
]
}
],
"source": [
"model.initialize(loss='BCE', seed=1, se_hPars=se_hPars.copy(), end='\\r')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Train for 50 epochs."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m\u001b[37mEpoch 9 - Batch 0/0 - Accuracy: 0.734 Cost: 0.57872 - TIME: 1.12s RATE: 8.93e+00e/s TTC: 0s \u001b[0m\n",
"\n",
"+-------+----------+----------+----------+-------+--------+-------+----------------------------------+\n",
"| \u001b[1m\u001b[37mepoch\u001b[0m | \u001b[1m\u001b[37mlrate\u001b[0m | \u001b[1m\u001b[37mlrate\u001b[0m | \u001b[1m\u001b[32maccuracy\u001b[0m | | \u001b[1m\u001b[31mBCE\u001b[0m | | \u001b[37mExperiment\u001b[0m |\n",
"| | \u001b[37mGRU\u001b[0m | \u001b[37mDense\u001b[0m | \u001b[1m\u001b[32mdtrain\u001b[0m | \u001b[1m\u001b[32mdval\u001b[0m | \u001b[1m\u001b[31mdtrain\u001b[0m | \u001b[1m\u001b[31mdval\u001b[0m | |\n",
"+-------+----------+----------+----------+-------+--------+-------+----------------------------------+\n",
"| \u001b[1m\u001b[37m0\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.583\u001b[0m | \u001b[1m\u001b[31m0.559\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m1\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.551\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m2\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.550\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m3\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m4\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m5\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m6\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m7\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m8\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"| \u001b[1m\u001b[37m9\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[37m5.00e-03\u001b[0m | \u001b[1m\u001b[32m0.734\u001b[0m | \u001b[1m\u001b[32m0.762\u001b[0m | \u001b[1m\u001b[31m0.579\u001b[0m | \u001b[1m\u001b[31m0.549\u001b[0m | \u001b[37m1635014424_GRU-1_Dense-2-softmax\u001b[0m |\n",
"+-------+----------+----------+----------+-------+--------+-------+----------------------------------+\n"
]
}
],
"source": [
"model.train(epochs=10, init_logs=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot the results."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtNElEQVR4nO3de3wV1bn/8c+TEEgQEORiy6UmIsglGIQAnlotR+qtrVhRQeul2CPU9qBHtB7UehQv9WdFa+1PW29F9CcWjvToQau1tUDVeg1WK0G8AUrwAiIgkQRI8vz+mEmYJJOdHchm5/J9v9iv7Fmz1ppnT8J+Zs3MXtvcHRERkboy0h2AiIi0TEoQIiISSwlCRERiKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShIi0eGZ2ipmtM7NSMzs83fG0F0oQrYCZzTCzIjPbYWbz6qzrbGa/MbPPzGyrmT0bWTfTzFab2Rdm9pGZ3WZmHSLr15pZWfifrtTM/lyn75lm9knYfq6ZdYqsu97M3jSzCjObnSD2uWbmZnZIzLpBZlZuZg9Fyr5jZs+b2ZZw2/eZWdeYtgeY2UYze77xPQhm1tXMfhm+5i/N7EMzW2Rm4yJ1PFxXambrw/qZdfbXt+r0OzVRDGY22cxeMLPtZrYsyVhzw1iqfy+fmtkTZnZsMu33JTO7xczeNbNtZrbKzM5N0aZuAWa4exdgc7h/OjTWSPaOEkTr8BFwAzA3Zt09wAHA0PDnzMi6xcAod+8G5AMFwEV12p/k7l3Cx3HVhWZ2PHA5MAE4CDgYuDbS7j3gP4E/NhS0mX0DGJjgdd0JvFqnbH+C19o3fE39gDkxbX8BvJWg72gcnYAlwAjgu0C3sO8FwIl1qheEb0LfBKYAP0xmGwl8DvwKuGkP2nYPYykA/gI8amZT9zKe5vYlcBLB7+0HwO1m9vUUbOcgoDgF/Uoi7q5HK3kQvHHOiywPAb4AuiXRtifwDPCbSNla4FsN1H8YuDGyPAH4JKbeQ8DsmPIOwD+AwwAHDqmz/gzgv4HZwEMJ4p4EvFmn7OvAi8B5wPNJvPbzgY+B/RqpVyvOML47E+0vYGoTYliW5O85N4ylQ53ynwKfAhnhcl/gD8BGYA1wUaTu7DD+B4FtBG+uhZH1s4D14bq3gQlheQbBgcH7wKawjwOa8De6GLg0wfqpwOpwu2uAsyLbvQr4ANgQxr0/0AkoDffHl2FcH4bLpeHjX8J+/w7cBmwJt/H1sHxd2OcPInF8J/z7/CJcPzuybkoYW7dw+UTgE6B3c/1fbi0PjSBat7EE/6GuDU8xvWlmp0YrmNn3zewL4DOCI9G76/QxPzxV82czK4iUDwfeiCy/ARxoZj2TjG0m8Ky7/7PuCjPrBlwHXJJEP0cTOXIMT/ncAcwgeJNIxreAp939yyTrY2ZDgKMIRkotxf8AfYBDzSwDeJzg99KPIIFfHI78qk0kGCV1J3jjvgPAzA4l2H9j3L0rcDxB8gO4EPgewQiqL7CZYKTXKDPLAcbQwJG+me0H/Bo4Mdzu14HXw9VTw8e/EoxWuwB3uPsOD0ZREIzuBhL8TUA4wnL3F8PlccA/CQ6GHg5f+xjgEOBs4A4zq+7rS+DccN98B/ixmX0PwN0XAi8Avw7/3n8HnO/uG5PZD22JEkTr1p/g1NFWgv/MM4AHzGxodQV3f9iDU0yDgbsIjkCrnUVwtHoQsBR42sy6h+u6hP1Wq35e73pAXWY2APgRcHUDVa4HfufuJY30cyzBaYtoPxcBL7v78sbiiOhFcARY3e/I8BrHF2b2dp26r5nZlwSnr5YBv2nCdlLto/DnAQRvfL3d/Tp33+nuq4F7CUZm1Z539yfdvRL4fwQHCACVBEfmw8wsy93Xuvv74boLgJ+5e4m77yAYiZyW5Pn+uwgS1tMJ6lQB+WaW4+4fu3t1MjkL+KW7r3b3UuAK4IwmXmdY4+73h693ITAAuC5MMn8GdhIkC9x9mbu/6e5V4UHM7wmSYrV/B44h+Bt43N2faEIcbYYSROtWBuwCbgjfJP5G8EZ/XN2K7v4uwZHdbyJlf3f3Mnff7u7/h2BoflS4upTgXH216ufbkojrVwT/MbfWXWFmIwmO6G9L1IGZHUFwFHiau78TlvUlSBA/SyKGqE3AV6sX3P11d+9OcPqqU526owiS4xSCI9L9IusqgKw69bMIfgeY2V2RC8tXNjHGZPQLf35OkNT7holui5ltAa4EDozU/yTyfDuQbWYd3P094GKCN/8NZrYg3LeE/T4a6fMtgoRyYKLXZ2ZzCA5WJnt4XqZu/XAEN4UgCX1sZn8MR2oQHOB8EOnyA4LTlNHX05jowU8ZgLvXLesSxjbOzJaGo+etYUy9qiu6+xbgkfA13dqEGNoUJYjWrd7pGxKfdulA4ovGDlj4vJjdR5yEzz91901JxDUBmBPehVT9JvWimX0fGE8wavkwXPdT4FQze626sQW3MS4Gfujuf430O5bgjX5l2PZ2YGy4nUwa9lfguPAUR6M88N8E1zmio5cPw9ij8gjf2Nz9At99wf/GZLbVRKcQnEt/m+C8+Rp37x55dHX3byfTUTiy/AZBQnCCi/6E/Z5Yp99sd1/f0Oszs2sJztMf5+5fRLZRr767P+3uxxL8HlcRjHogGB0dFAnxawQJOfoGX9N1Mq+xEQ8T/I0NcPf9CUY/1X/71QcyPyQYWfy6GbbXKilBtAJm1sHMsoFMINPMssOh97MEb1pXhHWOJDiH+3TY7nwz6xM+H0YwbP9ruPw1MzvSzDqG/V1GcAT193CzDwL/ZmbDwtNOVwHzIjFlhTFlAB3CPqrfpAcTJJSR4QOCO10eJbjramBk3V0Ed0IdH/abD/wJuNDdH6+zK54ieIOubns1wYXGkeFphYY8SHCR+lEzyzezzDD2wgRtILjzaJqZfSVcXkhwnn+IBQoJ3kQWNNRBZFsdgIxwP9UdhSRkZgea2QzgGuAKd68CXgG2mdksM8sJt5NvZmOS6O9QMzsmvLurnODIuipcfRfwczM7KKzb28xOTtDXFcD3CS7eJzx4CF/HyWGi3kEwSq3e7u+BmWaWF14nuBFY6O4VMV1tDNsd3NhrTaAr8Lm7l5vZ2PA1VMeZTXDzxZUEN0L0M7Of7MW2Wq90XBnXo2kPglMBXucxO1w3nOBI90tgJXBKpN39BEdgXxJchJwDZEfa/TNct4kgcRTW2e4lYfsvwr46RdbNi4lpagPx17uLqc5re6hOzFXsvkOlFChuoO1UkriDKKy7P8Gprw/C1/wBwR1AYxPFSZCUbg2fV9/h8264T1YC/9bIdqfG7Kd5jbTJZfddOl8SjBqeBE6oU68vwRvrJwQXk18ivMsqZr9W99mB4M6yVwhOF34OPAH0jbzGSwhGKdsI7hq6MUGszu43++rHlQ3U/SrwN4LrWVsIzu8Pi2z3aoIRzEaCN+geDf1uCG5y2Bj2c0TdvwWCaw1eZ/slwDfC56eFfwPbwtd/R/X+Ijj9+VSkXUG4nwal+71gXz8s3AEiIiK16BSTiIjEUoKQNsHMrozcMRN9PJXu2OKY2VkNxKtPC0uLoVNMIiISq81MdtWrVy/Pzc1NdxgiIq3K8uXLP3P33nHr2kyCyM3NpaioKN1hiIi0Kmb2QUPrdA1CRERiKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWG3mcxB75anL4ZM30x2FiMie+coIOPGmZu9WIwgREYmlEQSkJPOKiLR2GkGIiEgsJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWEoQIiISK6UJwsxOMLO3zew9M7s8Zv1tZvZ6+HjHzLZE1lVG1i1OZZwiIlJfyj4oZ2aZwJ3AsUAJ8KqZLXb3ldV13H1mpP6FwOGRLsrcfWSq4hMRkcRSOYIYC7zn7qvdfSewADg5Qf0zgd+nMB4REWmCVCaIfsC6yHJJWFaPmR0E5AFLIsXZZlZkZi+Z2fcaaDc9rFO0cePGZgpbRESg5VykPgNY5O6VkbKD3L0Q+D7wKzMbWLeRu9/j7oXuXti7d+99FauISLuQygSxHhgQWe4flsU5gzqnl9x9ffhzNbCM2tcnREQkxVKZIF4FBplZnpl1JEgC9e5GMrMhQA/gxUhZDzPrFD7vBRwJrKzbVkREUidldzG5e4WZzQCeBjKBue5ebGbXAUXuXp0szgAWuLtHmg8F7jazKoIkdlP07icREUk9q/2+3HoVFhZ6UVFRusMQEWlVzGx5eL23npZykVpERFoYJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWEoQIiISK2UflGtNrn28mJUffZHuMERE9siwvt245qThzd6vRhAiIhJLIwhISeYVEWntNIIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGKlNEGY2Qlm9raZvWdml8esv83MXg8f75jZlsi6H5jZu+HjB6mMU0RE6uuQqo7NLBO4EzgWKAFeNbPF7r6yuo67z4zUvxA4PHx+AHANUAg4sDxsuzlV8YqISG2pHEGMBd5z99XuvhNYAJycoP6ZwO/D58cDf3H3z8Ok8BfghBTGKiIidaQyQfQD1kWWS8KyeszsICAPWNKUtmY23cyKzKxo48aNzRK0iIgEWspF6jOARe5e2ZRG7n6Puxe6e2Hv3r1TFJqISPuUygSxHhgQWe4flsU5g92nl5raVkREUiCVCeJVYJCZ5ZlZR4IksLhuJTMbAvQAXowUPw0cZ2Y9zKwHcFxYJiIi+0jK7mJy9wozm0Hwxp4JzHX3YjO7Dihy9+pkcQawwN090vZzM7ueIMkAXOfun6cqVhERqc8i78utWmFhoRcVFaU7DBGRVsXMlrt7Ydy6lnKRWkREWhglCBERiaUEISIisZQgREQkVsruYhKRtmPXrl2UlJRQXl6e7lBkD2VnZ9O/f3+ysrKSbqMEISKNKikpoWvXruTm5mJm6Q5Hmsjd2bRpEyUlJeTl5SXdTqeYRKRR5eXl9OzZU8mhlTIzevbs2eQRoBKEiCRFyaF125PfnxKEiLQ6s2fP5pZbbgFg3rx5fPTRR03u46677uLBBx9s7tDaFF2DEJFWbd68eeTn59O3b9966yorK8nMzIxtd8EFF6Q6tD1WUVFBhw7pf3vWCEJEWoWf//znDB48mG984xu8/fbbACxatIiioiLOOussRo4cSVlZGbm5ucyaNYtRo0bxyCOPcO+99zJmzBgKCgo49dRT2b59O1B7FDJ+/HhmzZrF2LFjGTx4MM8991y97ZeWljJhwgRGjRrFiBEj+N///d+adQ8++CCHHXYYBQUFnHPOOQB8+umnnHLKKRQUFFBQUMALL7zA2rVryc/Pr2l3yy23MHv27JoYLr74YgoLC7n99tt5/PHHGTduHIcffjjf+ta3+PTTT2viOO+88xgxYgSHHXYYf/jDH5g7dy4XX3xxTb/33nsvM2fWfGHnHkt/ihKRVuXax4tZ+dEXzdrnsL7duOak4Q2uX758OQsWLOD111+noqKCUaNGMXr0aE477TTuuOMObrnlFgoLd08n1LNnT1577TUANm3axLRp0wC46qqr+N3vfseFF15YbxsVFRW88sorPPnkk1x77bU888wztdZnZ2fz6KOP0q1bNz777DOOOOIIJk6cyMqVK7nhhht44YUX6NWrF59/HswretFFF/HNb36TRx99lMrKSkpLS9m8OfG3Ju/cuZPqOeU2b97MSy+9hJlx3333cfPNN3Prrbdy/fXXs//++/Pmm2/W1MvKyuLnP/85c+bMISsri/vvv5+77767sd3eKCUIEWnxnnvuOU455RQ6d+4MwMSJExPWnzJlSs3zFStWcNVVV7FlyxZKS0s5/vjjY9tMmjQJgNGjR7N27dp6692dK6+8kmeffZaMjAzWr1/Pp59+ypIlSzj99NPp1asXAAcccAAAS5YsqbnGkZmZyf77799ogojGXVJSwpQpU/j444/ZuXNnze2pzzzzDAsWLKip16NHDwCOOeYYnnjiCYYOHcquXbsYMWJEwm0lQwlCRJok0ZF+S7HffvvVPJ86dSqPPfYYBQUFzJs3j2XLlsW26dSpExC8mVdUVNRbP3/+fDZu3Mjy5cvJysoiNze3ybeNdujQgaqqqprluu2jcV944YVccsklTJw4kWXLltWcimrI+eefz4033siQIUM477zzmhRXQ3QNQkRavKOPPprHHnuMsrIytm3bxuOPP16zrmvXrmzbtq3Bttu2beOrX/0qu3btYv78+Xscw9atW+nTpw9ZWVksXbqUDz74AAiO3B955BE2bdoEUHOKacKECfz2t78FgovlW7du5cADD2TDhg1s2rSJHTt28MQTTyTcXr9+/QB44IEHasqPPfZY7rzzzprl6lHJuHHjWLduHQ8//DBnnnnmHr/OKCUIEWnxRo0axZQpUygoKODEE09kzJgxNeumTp3KBRdcUHORuq7rr7+ecePGceSRRzJkyJA9juGss86iqKiIESNG8OCDD9b0NXz4cH72s5/xzW9+k4KCAi655BIAbr/9dpYuXcqIESMYPXo0K1euJCsri6uvvpqxY8dy7LHHJoxn9uzZnH766YwePbrm9BUE11E2b95Mfn4+BQUFLF26tGbd5MmTOfLII2tOO+0tfWGQiDTqrbfeYujQoekOQxrx3e9+l5kzZzJhwoTY9XG/R31hkIhIG7ZlyxYGDx5MTk5Og8lhT+gitYhIK9e9e3feeeedZu9XIwgREYmlBCEiIrGUIEREJJYShIiIxFKCEJFWJzrRXlNNnTqVRYsWNXNEbVPSCcLMOqcyEBGR9iBuGo+WqtEEYWZfN7OVwKpwucDMfpPyyEREIuKm+161ahVjx46tqbN27dqaSequu+46xowZQ35+PtOnT6exDwU3NC143LTdED/Fd93RSZcuXQBYtmwZRx11FBMnTmTYsGEAfO9732P06NEMHz6ce+65p6bNn/70J0aNGkVBQQETJkygqqqKQYMGsXHjRgCqqqo45JBDapZTKZnPQdwGHA8sBnD3N8zs6JRGJSIt11OXwydvNm+fXxkBJ97U4OqGpvseMmQIO3fuZM2aNeTl5bFw4cKaGVFnzJjB1VdfDcA555zDE088wUknndTgNiZNmhQ7LXjctN3FxcWxU3wn8tprr7FixYqaWVnnzp3LAQccQFlZGWPGjOHUU0+lqqqKadOm8eyzz5KXl8fnn39ORkYGZ599NvPnz+fiiy/mmWeeoaCggN69eye9e/dUUqeY3H1dnaLKFMQiIhIrOt13t27dak33PXnyZBYuXAhQK0EsXbqUcePGMWLECJYsWUJxcXHCbaxYsYKjjjqKESNGMH/+/Jr6S5Ys4cc//jGwe9ruhqb4TmTs2LE1yQHg17/+NQUFBRxxxBGsW7eOd999l5deeomjjz66pl51vz/84Q9rpg6fO3dus83W2phkRhDrzOzrgJtZFvAfwFupDUtEWqwER/rpMGXKFE4//XQmTZqEmTFo0CDKy8v5yU9+QlFREQMGDGD27NmNTs2d7LTgiUSn866qqmLnzp0166JTeS9btoxnnnmGF198kc6dOzN+/PiE8Q0YMIADDzyQJUuW8Morr+zVrLRNkcwI4gLg34F+wHpgZLgsIrJPJJrue+DAgWRmZnL99dfXjB6q32x79epFaWlpUnctNTQteNy03Q1N8Z2bm8vy5csBWLx4Mbt27Yrd1tatW+nRowedO3dm1apVvPTSSwAcccQRPPvss6xZs6ZWvxB838PZZ5/N6aef3uD3bDe3RhOEu3/m7me5+4Hu3sfdz3b3TfsiOBERSDzdNwSjiIceeojJkycDwdxE06ZNIz8/n+OPP75e/TgNTQseN213Q1N8T5s2jb/97W8UFBTw4osv1ho1RJ1wwglUVFQwdOhQLr/8co444ggAevfuzT333MOkSZMoKCio9Q1zEydOrPk+6n2l0em+zex+oF4ld/9hqoLaE5ruWyR1NN13+hUVFTFz5kyee+65Pe6jqdN9J3MNIvqVR9nAKcBHexyhiIg0yU033cRvf/vbfXbtoVqjCcLd/xBdNrPfA8+nLCIREanl8ssv5/LLL9/n292TqTYGAX2aOxAREWlZGh1BmNk2gmsQFv78BJiV4rhERCTNkjnF1HVfBCIiIi1LgwnCzEYlaujurzXWuZmdANwOZAL3uXu9T9iY2WRgNsHo5A13/35YXglUf57/Q3efWLetiIikTqIRxK0J1jlwTKKOzSwTuBM4FigBXjWzxe6+MlJnEHAFcKS7bzaz6LWNMncf2Uj8ItIOzZ49my5duvDTn/6UefPmcdxxx9G3b98m9XHXXXfRuXNnzj333KTqr127lqFDh3LooYfi7uy3337cf//9HHrooQA89dRT/Nd//Rfbt2+nU6dOHHPMMdx6663Mnj2be++9t9bcScuWLaN79+5NijcdGkwQ7v6ve9n3WOA9d18NYGYLgJOBlZE604A73X1zuM0Ne7lNEWln5s2bR35+fmyCqKysbPBTxxdccEGTtzVw4EBef/11AO6++25uvPFGHnjgAVasWMGMGTP44x//yJAhQ6isrKw1Q+vMmTP56U9/2uTtpVtSdzGZWb6ZTTazc6sfSTTrB0Qn+SsJy6IGA4PN7O9m9lJ4SqpatpkVheXfayCu6WGdon0x9a2IpE/cdN+LFi2iqKiIs846i5EjR1JWVkZubi6zZs1i1KhRPPLIIw1O4x390qHx48cza9Ysxo4dy+DBg5P6MNoXX3xBjx49ALj55pv52c9+VvMJ7MzMzJoJ/lqzZO5iugYYDwwDngROJPgcxIPNtP1BYf/9gWfNbIS7bwEOcvf1ZnYwsMTM3nT396ON3f0e4B4IPkndDPGISCN+8covWPX5qmbtc8gBQ5g1tuGbIxua7vu0007jjjvu4JZbbqGwcPeHgXv27MlrrwWXSTdt2hQ7jXddFRUVvPLKKzz55JNce+21PPPMM/XqvP/++4wcOZJt27axfft2Xn75ZSCYCfbSSy9tMP7bbruNhx56CIAePXqwdOnSJPZK+iXzSerTgALgH+5+npkdCDyURLv1wIDIcv+wLKoEeNnddwFrzOwdgoTxqruvB3D31Wa2DDgceB8RaXei030Dtab7jhOdw2jFihVcddVVbNmyhdLSUo4//vjYNpMmTQJg9OjRrF27NrZO9BTTwoULmT59On/6058ajb+1nmJKJkGUu3uVmVWYWTdgA7Xf+BvyKjDIzPIIEsMZwPfr1HkMOBO438x6EZxyWm1mPYDt7r4jLD8SuDmpVyQiKZXoSL+liE6Sl+w03p06dQKC00PJfC3oxIkTaybOGz58OMuXL6egoGDvg29BGrwGYWZ3mtk3gFfMrDtwL7AceA14sbGO3b0CmAE8TfD9Ef/t7sVmdp2ZVaf/p4FN4VeaLgUuC2eKHQoUmdkbYflN0bufRKR9STTdd9euXdm2bVuDbRuaxntvPf/88wwcOBCAyy67jBtvvJF33nkHCL4L4q677mq2baVLohHEO8AcoC/wJfB7gltWu7n7P5Pp3N2fJLhuES27OvLcgUvCR7TOC8CIZLYhIm1fdLrvPn361Jq+e+rUqVxwwQXk5OTw4ov1j12rp/Hu3bs348aNS5hMGlN9DcLd6dixI/fddx8Ahx12GL/61a8488wz2b59O2bGd7/73Zp20WsQAI899hi5ubl7HMe+ksx03wcRnB46A8ghSBQPu/u7qQ8veZruWyR1NN1329DU6b6T+cKgD9z9F+5+OMH1gu8BzXsLg4iItDiNJggz62BmJ5nZfOAp4G1gUsojExGRtEo0F9OxBCOGbwOvAAuA6e7+5T6KTURE0ijRReorgIeBS6unwhARkfYj0VxMCSfjExGRtm1PvlFORETaASUIEWl1ohPtNdXUqVNZtGhRbHleXh4jR45kyJAhXHvttTXrSktL+dGPfsTAgQMZPXo048ePr5mHKTMzk5EjR9Y8brqp3tfetFrJTLUhItIuzJkzh9NOO43y8nKGDRvGueeeS15eHueffz55eXm8++67ZGRksGbNGlauDCZ3yMnJqZmfqa3RCEJEWoW46b5XrVrF2LFja+qsXbuWESOCSRiuu+46xowZQ35+PtOnT6exDwVHlZeXA8GcTu+//z4vv/wyN9xwAxkZwVtmXl4e3/nOd5rrpbVYGkGISJN8cuON7HireT8r22noEL5y5ZUNrm9ouu8hQ4awc+dO1qxZQ15eHgsXLqyZyXXGjBlcfXUws88555zDE088wUknnZQwjssuu4wbbriB9957j4suuog+ffrw0ksvMXLkyAa/eKisrIyRI0fWLF9xxRW1ZpNtzTSCEJEWLzrdd7du3WpN9z158mQWLlwIUCtBLF26lHHjxjFixAiWLFlCcXFxo9uZM2cOr7/+Op988gl//etfeeGFFxptU32KqfrRVpIDaAQhIk2U6Eg/HaZMmcLpp5/OpEmTMDMGDRpEeXk5P/nJTygqKmLAgAHMnj275rRRMrp06cL48eN5/vnnOfXUU3njjTcSfn1pW6URhIi0eImm+x44cCCZmZlcf/31NUfv1cmgV69elJaWxt61lEhFRQUvv/wyAwcOZODAgRQWFnLNNdfUXMdYu3Ytf/zjH5vp1bVcGkGISIuXaLpvCEYRl112GWvWrAGge/fuTJs2jfz8fL7yla/Uq9+Q6msQO3fuZMKECTXfMnffffdx6aWXcsghh5CTk0OvXr2YM2cOUP8axAknnNBmbnVtdLrv1kLTfYukjqb7bhuafbpvERFpn5QgREQklhKEiIjEUoIQkaS0leuV7dWe/P6UIESkUdnZ2WzatElJopVydzZt2kR2dnaT2uk2VxFpVP/+/SkpKWHjxo3pDkX2UHZ2Nv37929SGyUIEWlUVlYWeXl56Q5D9jGdYhIRkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhIiKxlCBERCSWEoSIiMRSghARkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhIiKxlCBERCRWShOEmZ1gZm+b2XtmdnkDdSab2UozKzazhyPlPzCzd8PHD1IZp4iI1JeyLwwys0zgTuBYoAR41cwWu/vKSJ1BwBXAke6+2cz6hOUHANcAhYADy8O2m1MVr4iI1JbKEcRY4D13X+3uO4EFwMl16kwD7qx+43f3DWH58cBf3P3zcN1fgBNSGKuIiNSRygTRD1gXWS4Jy6IGA4PN7O9m9pKZndCEtpjZdDMrMrMifVeuiEjzSvdF6g7AIGA8cCZwr5l1T7axu9/j7oXuXti7d+/URCgi0k6lMkGsBwZElvuHZVElwGJ33+Xua4B3CBJGMm1FRCSFUpkgXgUGmVmemXUEzgAW16nzGMHoATPrRXDKaTXwNHCcmfUwsx7AcWGZiIjsIym7i8ndK8xsBsEbeyYw192Lzew6oMjdF7M7EawEKoHL3H0TgJldT5BkAK5z989TFauIiNRn7p7uGJpFYWGhFxUVpTsMEZFWxcyWu3th3Lp0X6QWEZEWSglCRERiKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWO0+Qbg7q7eupnRnKW3lQ4MiIs0hZVNttBZbdmzh5MeCr6nI6ZBD75ze9O7cmz45fYKfnfvsLgufd87qnOaoRURSr90niE6ZnfjFUb9gY9lGNmzfwMbtG9lQtoHiTcVsWLeB8sryem26ZHWplUQaSiidMjul4RWJiDSPdp8gOmd15tsHfzt2nbtTuqu0Jmls3B4mkUgy+ceGf7Bh+wZ2Ve2q175bx26xI5A+nfvUJJVeOb3IysxK9csUEWmydp8gEjEzunbsSteOXTm4+8EN1nN3tu7YWi+JbNy+sebnmk/W8Nn2z6jwinrtD8g+gN45vemZ05OsjCzMjEzLJMMyaj0yLRPDyMwIfzZUp4H21evrlZNBRsbu/jMsAzPbvR+wejFH91FD9aLLCfszGlyXaNvJim57j9o3QwzNoaXE0XLCaBmBtIQ4unXqxugDRzd7v0oQzcDM6J7dne7Z3RncY3CD9aq8is3lm+udztq4PUgim8o3UVFVQZVXUemVuHvwE6eyKvzplVRVVVFFFVVeVa9udVn1w9GFd5G27rBehzH/O/ObvV8liH0owzLomdOTnjk9GXLAkH2yTXevlUhqkgdVtRJNrQTkVZEO6vQXKaj1vM4dYA3VS9hfgj721N72oTvbamspBxwt5ffSUvZHdofslPSrBNHGVZ9uyiSTLHStQ0SS1+4/ByEiIvGUIEREJJYShIiIxFKCEBGRWEoQIiISSwlCRERiKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWEoQIiISSwlCRERiKUGIiEisdv99EL5zJx9O/xHZhx5Kdv5wsocPp2NuLpah3Cki7Vu7TxAVm7fgZWVsXrAA37EDgIzOnek0bCg5w4OEUZM0MjPTHK2IyL7T7hNE1oF9yF24AK+oYMf771O+opjy4uCxecHC2klj6FBy8pU0RKR9sJby3a57q7Cw0IuKipq1zyBprK5JGOXFxZSvWoWXlwNgnTuTPXQo2cOH1Yw2OublKWmISKthZsvdvTB2nRJE03hFBTtWr6a8eGXtpFFWBoRJY8iQcJQxjJz8fCUNEWmxlCBSzCsr2bl6NWXFxbtPUSVKGsOH0/Hgg5U0RCTt0pYgzOwE4HYgE7jP3W+qs34qMAdYHxbd4e73hesqgTfD8g/dfWKibaUzQcSplTSqRxtvvbU7aeTkRJJGkDg6HXww1qHdXxYSkX0oLQnCzDKBd4BjgRLgVeBMd18ZqTMVKHT3GTHtS929S7Lba2kJIo5XVrJzzRrKi4t3J4633sK3bwfAsrLI6NIFy8kmI6czGdnZZOTkYDk5ZOTkkJGTjWXnhGWROp1zwvKwfnYOGZ1zyMjOxnI6B+XZ2VjHjmneAyLS0iRKEKk8XB0LvOfuq8MgFgAnAysTtmrDLDOTToccQqdDDmH/k08GwqSxdi3lxcXsePddKktL8bJyqsrLqSrbjpeVU7llCxWffEzV9rKwvCwYiTQ1uXfoUD/pZGfXTkA5OWR0yobMzOCzIJmZWGYGZNT5mdkhvjz6s0Nm4vWJ6mVmglkQg1nwwIJ/1ctmkJEBWLjaaj8wLMNiyiN9VPffQB8WadPYT6teFmkjUpkg+gHrIsslwLiYeqea2dEEo42Z7l7dJtvMioAK4CZ3f6xuQzObDkwH+NrXvtaMoe87lplJp4ED6TRwYJPauTu+cydV27fjYdKoThxVZeVUlUeel4V1tpeF5WH98jI8TDq7tm4N65fh5eV4VRVUVtb7KU2QbGJpYv2a6nHbiqmX7LoG+2usjyaV1V3cm75azzaT3u4ebjP70EPp98tb69fbS+k+4f048Ht332FmPwIeAI4J1x3k7uvN7GBgiZm96e7vRxu7+z3APRCcYtqXgaebmWGdOpHRqdM+3W5NwqisjE0g9X7WrVdZBVV1flZWxJd7FbgHbZ1wxOTBT3fcPSivqqopd69ev7t+0D5SFunH3aEqpqymblX4wj384bWWqf6rq1vOntWvv5069YlZrrXO61TzhlY13Ee9kalHVsX8N4stq1+U8DUkaJjybe5VHElsM+myPd9mVv/+MYHsvVQmiPXAgMhyf3ZfjAbA3TdFFu8Dbo6sWx/+XG1my4DDgVoJQvY9y8iAjAwsKyvdoYhIiqVywqFXgUFmlmdmHYEzgMXRCmb21cjiROCtsLyHmXUKn/cCjqQdX7sQEUmHlI0g3L3CzGYATxPc5jrX3YvN7DqgyN0XAxeZ2USC6wyfA1PD5kOBu82siiCJ3RS9+0lERFJPH5QTEWnHEt3mqjmtRUQklhKEiIjEUoIQEZFYShAiIhJLCUJERGK1mbuYzGwj8MFedNEL+KyZwmnttC9q0/6oTftjt7awLw5y995xK9pMgthbZlbU0K1e7Y32RW3aH7Vpf+zW1veFTjGJiEgsJQgREYmlBLHbPekOoAXRvqhN+6M27Y/d2vS+0DUIERGJpRGEiIjEUoIQEZFY7T5BmNkJZva2mb1nZpenO550MrMBZrbUzFaaWbGZ/Ue6Y0o3M8s0s3+Y2RPpjiXdzKy7mS0ys1Vm9paZ/Uu6Y0onM5sZ/j9ZYWa/N7PsdMfU3Np1gjCzTOBO4ERgGHCmmQ1Lb1RpVQFc6u7DgCOAf2/n+wPgPwi/yEq4HfiTuw8BCmjH+8XM+gEXAYXunk/wnTdnpDeq5teuEwQwFnjP3Ve7+05gAXBymmNKG3f/2N1fC59vI3gD6JfeqNLHzPoD3yH4Otx2zcz2B44Gfgfg7jvdfUtag0q/DkCOmXUAOgMfpTmeZtfeE0Q/YF1kuYR2/IYYZWa5BN8D/nKaQ0mnXwH/CVSlOY6WIA/YCNwfnnK7z8z2S3dQ6eLu64FbgA+Bj4Gt7v7n9EbV/Np7gpAYZtYF+ANwsbt/ke540sHMvgtscPfl6Y6lhegAjAJ+6+6HA18C7faanZn1IDjbkAf0BfYzs7PTG1Xza+8JYj0wILLcPyxrt8wsiyA5zHf3/0l3PGl0JDDRzNYSnHo8xsweSm9IaVUClLh79YhyEUHCaK++Baxx943uvgv4H+DraY6p2bX3BPEqMMjM8sysI8FFpsVpjiltzMwIzjG/5e6/THc86eTuV7h7f3fPJfi7WOLube4IMVnu/gmwzswODYsmACvTGFK6fQgcYWadw/83E2iDF+07pDuAdHL3CjObATxNcBfCXHcvTnNY6XQkcA7wppm9HpZd6e5Ppi8kaUEuBOaHB1OrgfPSHE/auPvLZrYIeI3g7r9/0Aan3dBUGyIiEqu9n2ISEZEGKEGIiEgsJQgREYmlBCEiIrGUIEREJJYShEgTmFmlmb0eeTTbp4nNLNfMVjRXfyJ7q11/DkJkD5S5+8h0ByGyL2gEIdIMzGytmd1sZm+a2StmdkhYnmtmS8zsn2b2VzP7Wlh+oJk9amZvhI/qaRoyzeze8HsG/mxmOWl7UdLuKUGINE1OnVNMUyLrtrr7COAOgplgAf4v8IC7HwbMB34dlv8a+Ju7FxDMaVT9Cf5BwJ3uPhzYApya0lcjkoA+SS3SBGZW6u5dYsrXAse4++pwwsNP3L2nmX0GfNXdd4XlH7t7LzPbCPR39x2RPnKBv7j7oHB5FpDl7jfsg5cmUo9GECLNxxt43hQ7Is8r0XVCSSMlCJHmMyXy88Xw+Qvs/irKs4Dnwud/BX4MNd97vf++ClIkWTo6EWmanMhMtxB8R3P1ra49zOyfBKOAM8OyCwm+he0ygm9kq54B9T+Ae8zs3whGCj8m+GYykRZD1yBEmkF4DaLQ3T9LdywizUWnmEREJJZGECIiEksjCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFY/x9Y8w77x8QzCQAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"model.plot(path=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Overall, and using this dummy dataset made of string features, there is no significant metrics/cost difference from the simple Perceptron to recurrent RNN, GRU and LSTM. In this situation, one would favor the simple Perceptron because it computes faster. At least, it is important to note that the best architecture is not the fanciest, but simply the one that suits your needs and resources."
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"For code, maths and pictures behind the *GRU* layer, follow this link:\n",
"\n",
"* [Gated Recurrent Unit (GRU)](https://epynn.net/GRU.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Write, read & Predict"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A trained model can be written on disk such as:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m\u001b[32mMake: /media/synthase/beta/EpyNN/epynnlive/dummy_string/models/1635014424_GRU-1_Dense-2-softmax.pickle\u001b[0m\n"
]
}
],
"source": [
"model.write()\n",
"\n",
"# model.write(path=/your/custom/path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A model can be read from disk such as:"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"model = read_model()\n",
"\n",
"# model = read_model(path=/your/custom/path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can retrieve new features and predict on them."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"X_features, _ = prepare_dataset(N_SAMPLES=10)\n",
"\n",
"dset = model.predict(X_features, X_encode=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Results can be extracted such as:"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0 1 [0.27207978 0.72792022]\n",
"1 1 [0.26511927 0.73488073]\n",
"2 1 [0.26579407 0.73420593]\n",
"3 1 [0.26865469 0.73134531]\n",
"4 1 [0.26554721 0.73445279]\n",
"5 1 [0.2699316 0.7300684]\n",
"6 1 [0.26448517 0.73551483]\n",
"7 1 [0.26460705 0.73539295]\n",
"8 1 [0.26701639 0.73298361]\n",
"9 1 [0.26556502 0.73443498]\n"
]
}
],
"source": [
"for n, pred, probs in zip(dset.ids, dset.P, dset.A):\n",
" print(n, pred, probs)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}