CNTK/Tutorials/CNTK_207_Training_with_Sampled_Softmax.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNTK 207: Sampled Softmax\n",
    "\n",
    "For classification and prediction problems a typical criterion function is cross-entropy with softmax. If the number of output classes is high the computation of this criterion and the corresponding gradients could be quite costly. Sampled Softmax is a heuristic to speed up training in these cases. (see: [Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model](http://www.iro.umontreal.ca/~lisa/pointeurs/importance_samplingIEEEtnn.pdf), [Exploring the Limits of Language Modeling](https://arxiv.org/pdf/1602.02410v1.pdf), [What is Candidate Sampling](https://www.tensorflow.org/extras/candidate_sampling.pdf))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Select the notebook runtime environment devices / settings**\n",
    "\n",
    "Before we dive into the details we run some setup that is required for automated testing of this notebook. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)\n",
    "from __future__ import division\n",
    "\n",
    "import os\n",
    "import cntk as C\n",
    "import cntk.tests.test_utils\n",
    "cntk.tests.test_utils.set_device_from_pytest_env() # (only needed for our build system)\n",
    "C.cntk_py.set_fixed_random_seed(1) # fix a random seed for CNTK components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic concept\n",
    "\n",
    "The softmax function is used in neural networks if we want to interpret the network output as a probability distribution over a set of classes $C$ with $|C|=N_C$.\n",
    "\n",
    "Softmax maps an $N_C$-dimensional vector $z$, which has unrestricted values, to an $N_C$ dimensional vector $p$ with non-negative values that sum up to 1 so that they can be interpreted as probabilities. More precisely:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "p_i &= softmax(z, i)\\\\\n",
    "    &= \\frac{exp(z_i)}{\\sum_{k\\in C} exp(z_k)}\\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "In what follows we assume that the input $z$ to the softmax is computed from some hidden vector $h$ of dimension $N_h$  in a specific way, namely:\n",
    "\n",
    "$$ z = W h + b $$\n",
    "\n",
    "where $W$ is a learnable weight matrix of dimension $(N_c, N_h)$ and $b$ is a learnable bias vector.\n",
    "We restrict ourselves to this specific choice of $z$ because it helps in implementing an efficient sampled softmax.\n",
    "\n",
    "In a typical use-case like for example a recurrent language model, the hidden vector $h$ would be the output of the recurrent layers and $C$ would be the set of words to predict.   \n",
    "\n",
    "As a training criterion, we use cross-entropy which is a function of the expected (true) class $t\\in C$ and the probability predicted for it:\n",
    "\n",
    "$$cross\\_entropy := -log(p_t)$$\n",
    "\n",
    "## Sampled Softmax\n",
    "\n",
    "For the normal softmax the CNTK Python-api provides the function [cross_entropy_with_softmax](https://cntk.ai/pythondocs/cntk.ops.html?highlight=softmax#cntk.ops.cross_entropy_with_softmax). This takes as input the $N_C$-dimensional vector $z$. As mentioned for our sampled softmax implementation we assume that this z is computed by $ z = W h + b $. In sampled softmax this has to be part of the whole implementation of the criterion.\n",
    "\n",
    "Below we show the code for `cross_entropy_with_sampled_softmax_and_embedding`. Let’s look at the signature first.\n",
    "\n",
    "One fundamental difference to the corresponding function in the Python-api (`cross_entropy_with_softmax`) is that in the Python api function the input corresponds to $z$ and must have the same dimension as the target vector, while in cross_entropy_with_full_softmax the input corresponds to our hidden vector $h$ can have any dimension (hidden_dim).\n",
    "Actually, hidden_dim will be typically much lower than the dimension of the target vector.\n",
    "\n",
    "We also have some additional parameters `num_samples, sampling_weights, allow_duplicates` that control the random sampling. \n",
    "Another difference to the api function is that we return a triple (z, cross_entropy_on_samples, error_on_samples).\n",
    "\n",
    "We will come back to the details of the implementation below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Creates a subgraph computing cross-entropy with sampled softmax.\n",
    "def cross_entropy_with_sampled_softmax_and_embedding(\n",
    "    hidden_vector,            # Node providing hidden input\n",
    "    target_vector,            # Node providing the expected labels (as sparse vectors)\n",
    "    num_classes,              # Number of classes\n",
    "    hidden_dim,               # Dimension of the hidden vector\n",
    "    num_samples,              # Number of samples to use for sampled softmax\n",
    "    sampling_weights,         # Node providing weights to be used for the weighted sampling\n",
    "    allow_duplicates = True,  # Boolean flag to control whether to use sampling with replacemement \n",
    "                              # (allow_duplicates == True) or without replacement.\n",
    "    ):\n",
    "    # define the parameters learnable parameters\n",
    "    b = C.Parameter(shape = (num_classes, 1), init = 0)\n",
    "    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.glorot_uniform())\n",
    "\n",
    "    # Define the node that generates a set of random samples per minibatch\n",
    "    # Sparse matrix (num_samples * num_classes)\n",
    "    sample_selector = C.random_sample(sampling_weights, num_samples, allow_duplicates)\n",
    "\n",
    "    # For each of the samples we also need the probablity that it in the sampled set.\n",
    "    inclusion_probs = C.random_sample_inclusion_frequency(sampling_weights, num_samples, allow_duplicates) # dense row [1 * vocab_size]\n",
    "    log_prior = C.log(inclusion_probs) # dense row [1 * num_classes]\n",
    "\n",
    "    # Create a submatrix wS of 'weights\n",
    "    W_sampled = C.times(sample_selector, W) # [num_samples * hidden_dim]\n",
    "    z_sampled = C.times_transpose(W_sampled, hidden_vector) + C.times(sample_selector, b) - C.times_transpose (sample_selector, log_prior)# [num_samples]\n",
    "\n",
    "    # Getting the weight vector for the true label. Dimension hidden_dim\n",
    "    W_target = C.times(target_vector, W) # [1 * hidden_dim]\n",
    "    z_target = C.times_transpose(W_target, hidden_vector) + C.times(target_vector, b) - C.times_transpose(target_vector, log_prior) # [1]\n",
    "\n",
    "\n",
    "    z_reduced = C.reduce_log_sum_exp(z_sampled)\n",
    "    \n",
    "    # Compute the cross entropy that is used for training.\n",
    "    # We don't check whether any of the classes in the random samples conincides with the true label, so it might\n",
    "    # happen that the true class is counted\n",
    "    # twice in the normalising demnominator of sampled softmax.\n",
    "    cross_entropy_on_samples = C.log_add_exp(z_target, z_reduced) - z_target\n",
    "\n",
    "    # For applying the model we also output a node providing the input for the full softmax\n",
    "    z = C.times_transpose(W, hidden_vector) + b\n",
    "    z = C.reshape(z, shape = (num_classes))\n",
    "\n",
    "    zSMax = C.reduce_max(z_sampled)\n",
    "    error_on_samples = C.less(z_target, zSMax)\n",
    "    return (z, cross_entropy_on_samples, error_on_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To give a better idea of what the inputs and outputs are and how this all differs from the normal softmax we give below a corresponding function using normal softmax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Creates subgraph computing cross-entropy with (full) softmax.\n",
    "def cross_entropy_with_softmax_and_embedding(\n",
    "    hidden_vector,  # Node providing hidden input\n",
    "    target_vector,  # Node providing the expected labels (as sparse vectors)\n",
    "    num_classes,    # Number of classes\n",
    "    hidden_dim      # Dimension of the hidden vector\n",
    "    ):\n",
    "    # Setup bias and weights\n",
    "    b = C.Parameter(shape = (num_classes, 1), init = 0)\n",
    "    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.glorot_uniform())\n",
    "\n",
    "    \n",
    "    z = C.reshape( C.times_transpose(W, hidden_vector) + b, (1, num_classes))\n",
    "    \n",
    "    # Use cross_entropy_with_softmax\n",
    "    cross_entropy = C.cross_entropy_with_softmax(z, target_vector)\n",
    "\n",
    "    zMax = C.reduce_max(z)\n",
    "    zT = C.times_transpose(z, target_vector)\n",
    "    error_on_samples = C.less(zT, zMax)\n",
    "\n",
    "    return (z, cross_entropy, error_on_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see the main differences to the api function `cross_entropy_with_softmax` are:\n",
    "* We include the mapping $ z = W h + b $ into the function.\n",
    "* We return a triple (z, cross_entropy, error_on_samples) instead of just returning the cross entropy.\n",
    "\n",
    "\n",
    "## A toy example\n",
    "\n",
    "To explain how to integrate sampled softmax let us look at a toy example. In this toy example we first transform one-hot input vectors via some random projection into a lower dimensional vector $h$. The modeling task is to reverse this mapping using (sampled) softmax. Well, as already said this is a toy example.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start...\n",
      "Learning rate per 1 samples: 0.03\n",
      "Momentum per 100 samples: 0.8187307530779818\n",
      "\n",
      "Minbatch=5 Cross-entropy from full softmax = 3.844 perplexity = 46.705 samples/s = 9723.4\n",
      " Minibatch[   1-  10]: loss = 2.332016 * 1000, metric = 78.10% * 1000;\n",
      "\n",
      "Minbatch=15 Cross-entropy from full softmax = 3.467 perplexity = 32.042 samples/s = 9497.0\n",
      " Minibatch[  11-  20]: loss = 2.038090 * 1000, metric = 58.90% * 1000;\n",
      "\n",
      "Minbatch=25 Cross-entropy from full softmax = 3.067 perplexity = 21.484 samples/s = 8569.1\n",
      " Minibatch[  21-  30]: loss = 1.704383 * 1000, metric = 34.30% * 1000;\n",
      "\n",
      "Minbatch=35 Cross-entropy from full softmax = 2.777 perplexity = 16.074 samples/s = 10858.4\n",
      " Minibatch[  31-  40]: loss = 1.471806 * 1000, metric = 28.40% * 1000;\n",
      "\n",
      "Minbatch=45 Cross-entropy from full softmax = 2.459 perplexity = 11.688 samples/s = 10747.2\n",
      " Minibatch[  41-  50]: loss = 1.231299 * 1000, metric = 10.50% * 1000;\n",
      "\n",
      "Minbatch=55 Cross-entropy from full softmax = 2.269 perplexity = 9.671 samples/s = 10193.6\n",
      " Minibatch[  51-  60]: loss = 1.045713 * 1000, metric = 6.70% * 1000;\n",
      "\n",
      "Minbatch=65 Cross-entropy from full softmax = 2.104 perplexity = 8.203 samples/s = 10609.4\n",
      " Minibatch[  61-  70]: loss = 0.974900 * 1000, metric = 7.00% * 1000;\n",
      "\n",
      "Minbatch=75 Cross-entropy from full softmax = 1.937 perplexity = 6.941 samples/s = 10206.4\n",
      " Minibatch[  71-  80]: loss = 0.901665 * 1000, metric = 5.10% * 1000;\n",
      "\n",
      "Minbatch=85 Cross-entropy from full softmax = 1.798 perplexity = 6.036 samples/s = 10735.0\n",
      " Minibatch[  81-  90]: loss = 0.781453 * 1000, metric = 1.60% * 1000;\n",
      "\n",
      "Minbatch=95 Cross-entropy from full softmax = 1.707 perplexity = 5.514 samples/s = 11769.0\n",
      "done.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from math import log, exp, sqrt\n",
    "from cntk.logging import ProgressPrinter\n",
    "import timeit\n",
    "\n",
    "# A class with all parameters\n",
    "class Param:\n",
    "    # Learning parameters\n",
    "    learning_rate = 0.03\n",
    "    minibatch_size = 100\n",
    "    num_minbatches = 100\n",
    "    test_set_size = 1000\n",
    "    momentum = 0.8187307530779818\n",
    "    reporting_interval = 10\n",
    "    allow_duplicates = False\n",
    "    \n",
    "    # Parameters for sampled softmax\n",
    "    use_sampled_softmax = True\n",
    "    use_sparse = True\n",
    "    softmax_sample_size = 10\n",
    "\n",
    "    # Details of data and model\n",
    "    num_classes = 50\n",
    "    hidden_dim = 10\n",
    "    \n",
    "data_sampling_distribution = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)\n",
    "    \n",
    "softmax_sampling_weights = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)\n",
    "\n",
    "# Creates random one-hot vectors of dimension 'num_classes'.\n",
    "# Returns a tuple with a list of one-hot vectors, and list with the indices they encode.\n",
    "def get_random_one_hot_data(num_vectors):\n",
    "    indices = np.random.choice(\n",
    "        range(Param.num_classes),\n",
    "        size=num_vectors, \n",
    "        p = data_sampling_distribution()).reshape((num_vectors, 1))\n",
    "    list_of_vectors = C.Value.one_hot(indices, Param.num_classes)\n",
    "    return (list_of_vectors, indices.flatten())\n",
    "\n",
    "# Create a network that:\n",
    "# * Transforms the input one hot-vectors with a constant random embedding\n",
    "# * Applies a linear decoding with parameters we want to learn\n",
    "def create_model(labels):\n",
    "    # random projection matrix\n",
    "    random_data = np.random.normal(scale = sqrt(1.0/Param.hidden_dim), size=(Param.num_classes, Param.hidden_dim)).astype(np.float32)\n",
    "    random_matrix = C.constant(shape = (Param.num_classes, Param.hidden_dim), value = random_data)\n",
    "    \n",
    "    h = C.times(labels, random_matrix)\n",
    "    \n",
    "    # Connect the latent output to (sampled/full) softmax.\n",
    "    if Param.use_sampled_softmax:\n",
    "        sampling_weights = np.asarray(softmax_sampling_weights(), dtype=np.float32)\n",
    "        sampling_weights.reshape((1, Param.num_classes))\n",
    "        softmax_input, ce, errs = cross_entropy_with_sampled_softmax_and_embedding(\n",
    "            h, \n",
    "            labels,\n",
    "            Param.num_classes, \n",
    "            Param.hidden_dim, \n",
    "            Param.softmax_sample_size, \n",
    "            softmax_sampling_weights(),\n",
    "            Param.allow_duplicates)\n",
    "    else:\n",
    "        softmax_input, ce, errs = cross_entropy_with_softmax_and_embedding(\n",
    "            h, \n",
    "            labels, \n",
    "            Param.num_classes, \n",
    "            Param.hidden_dim)\n",
    "\n",
    "    return softmax_input, ce, errs\n",
    "\n",
    "def train(do_print_progress):\n",
    "    labels = C.input_variable(shape = Param.num_classes, is_sparse = Param.use_sparse)\n",
    "    z, cross_entropy, errs = create_model(labels)\n",
    "\n",
    "    # Setup the trainer\n",
    "    learning_rate_schedule = C.learning_parameter_schedule_per_sample(Param.learning_rate)\n",
    "    momentum_schedule = C.momentum_schedule(Param.momentum, minibatch_size = Param.minibatch_size)\n",
    "    learner = C.momentum_sgd(z.parameters, learning_rate_schedule, momentum_schedule, True)\n",
    "    progress_writers = None\n",
    "    if do_print_progress:\n",
    "        progress_writers = [ProgressPrinter(freq=Param.reporting_interval, tag='Training')]\n",
    "    trainer = C.Trainer(z, (cross_entropy, errs), learner, progress_writers)\n",
    "\n",
    "    minbatch = 0\n",
    "    average_cross_entropy = compute_average_cross_entropy(z)\n",
    "    minbatch_data = [0] # store minibatch values\n",
    "    cross_entropy_data = [average_cross_entropy] # store cross_entropy values\n",
    "\n",
    "    # Run training\n",
    "    t_total= 0\n",
    "\n",
    "    # Run training\n",
    "    for minbatch in range(1,Param.num_minbatches):\n",
    "        # Specify the mapping of input variables in the model to actual minibatch data to be trained with\n",
    "        label_data, indices = get_random_one_hot_data(Param.minibatch_size)\n",
    "        arguments = ({labels : label_data})\n",
    "\n",
    "        # If do_print_progress is True, this will automatically print the progress using ProgressPrinter\n",
    "        # The printed loss numbers are computed using the sampled softmax criterion\n",
    "        t_start = timeit.default_timer()\n",
    "        trainer.train_minibatch(arguments)\n",
    "        t_end = timeit.default_timer()\n",
    "\n",
    "        t_delta = t_end - t_start\n",
    "        samples_per_second = Param.minibatch_size / t_delta\n",
    "        \n",
    "        # We ignore the time measurements of the first two minibatches\n",
    "        if minbatch > 2:\n",
    "            t_total += t_delta\n",
    "\n",
    "        # For comparison also print result using the full criterion\n",
    "        if minbatch % Param.reporting_interval == int(Param.reporting_interval/2):\n",
    "            # memorize the progress data for plotting\n",
    "            average_cross_entropy = compute_average_cross_entropy(z)\n",
    "            minbatch_data.append(minbatch)\n",
    "            cross_entropy_data.append(average_cross_entropy)\n",
    "            \n",
    "            if do_print_progress:\n",
    "                print(\"\\nMinbatch=%d Cross-entropy from full softmax = %.3f perplexity = %.3f samples/s = %.1f\"\n",
    "                    % (minbatch, average_cross_entropy, exp(average_cross_entropy), samples_per_second))\n",
    "                \n",
    "    # Number of samples we measured. First two minbatches were ignored\n",
    "    samples_measured = Param.minibatch_size * (Param.num_minbatches - 2)\n",
    "    overall_samples_per_second = samples_measured / t_total\n",
    "    return (minbatch_data, cross_entropy_data, overall_samples_per_second) \n",
    "\n",
    "def compute_average_cross_entropy(softmax_input):\n",
    "    vectors, indices = get_random_one_hot_data(Param.test_set_size)\n",
    "    total_cross_entropy = 0.0\n",
    "    arguments = (vectors)\n",
    "    z = softmax_input.eval(arguments).reshape(Param.test_set_size, Param.num_classes)\n",
    "\n",
    "    for i in range(len(indices)):\n",
    "        log_p = log_softmax(z[i], indices[i])\n",
    "        total_cross_entropy -= log_p\n",
    "\n",
    "    return total_cross_entropy / len(indices)\n",
    "\n",
    "# Computes log(softmax(z,index)) for a one-dimensional numpy array z in an numerically stable way.\n",
    "def log_softmax(z,    # numpy array\n",
    "                index # index into the array\n",
    "            ):\n",
    "    max_z = np.max(z)\n",
    "    return z[index] - max_z - log(np.sum(np.exp(z - max_z)))\n",
    "\n",
    "\n",
    "\n",
    "np.random.seed(1)\n",
    "\n",
    "print(\"start...\")\n",
    "train(do_print_progress = True)\n",
    "print(\"done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above code we use two different methods to report training progress:\n",
    "1. Using a function that computes the average cross entropy on full softmax.\n",
    "2. Using the built-in ProgressPrinter\n",
    "\n",
    "ProgressPrinter reports how the value of the training criterion changes over time.\n",
    "In our case the training criterion is cross-entropy from **sampled** softmax.\n",
    "The same is true for the error rate computed by progress printer, this is computed only for true-class vs sampled-classes and will therefore underestimate the true error rate.\n",
    "\n",
    "Therefore while ProgressPrinter already gives us some idea how training goes on, if we want to compare the behavior for different sampling strategies (sample size, sampling weights, ...) we should not rely on numbers that are computed only using the sampled subset of classes. \n",
    "\n",
    "\n",
    "## Importance sampling\n",
    "\n",
    "Often the we don't have uniform distribution for the classes on the output side. The typical example is when we have words as output classes. A typical example are words where e.g. 'the' will be much more frequent than most others.\n",
    "\n",
    "In such cases one often uses a non uniform distribution for drawing the samples in sampled softmax but instead increases the sampling weight for the frequent classes. This is also called importane sampling.\n",
    "In our example the sampling distribution is controlled by the weight array `softmax_sampling_weights`.\n",
    "\n",
    "As an example let's look at the case where the classes are distrubted according to zipf-distrubtion like:\n",
    "$$\n",
    "p[i] \\propto \\frac{1}{i+5},\n",
    "$$\n",
    "actually we use this distribution already in our example.\n",
    "\n",
    "How does training behavior change if we switch uniform sampling to sampling with the zipfian distribution in sampled softmax?\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start...\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYVPX1x/H32aWJIKiAIAhIxEYR\nFVHUCKK/xIJYEwmgEgt2bImx9xh77CgKsWDs3WisKBoVpSiKWBBRaQooq0iT3fP749xlV9xlB9jZ\nu7vzeT3PPM7ce2fmDLPu2W87X3N3REREAPLSDkBERKoPJQUREVlBSUFERFZQUhARkRWUFEREZAUl\nBRERWUFJQUREVlBSEBGRFZQURERkhTppB7C6mjVr5u3bt087DBGRGmX8+PHz3L15RdfVuKTQvn17\nxo0bl3YYIiI1ipl9mcl16j4SEZEVsp4UzCzfzCaa2TNlnKtvZg+a2VQzG2tm7bMdj4iIlK8qWgqn\nAFPKOXcU8L27bwb8E7iyCuIREZFyZDUpmFkbYF/gznIu2R+4O7n/CLCHmVk2YxIRkfJlu6VwPXAm\nUFTO+dbA1wDuvhwoADbMckwiIlKOrCUFM+sLfOvu41d1WRnHfrXrj5kNMbNxZjZu7ty5lRajiIj8\nUjZbCrsA/cxsOvAA0MfMRq10zQxgEwAzqwM0Ab5b+YXcfbi7d3f37s2bVzjNVkRE1lDWkoK7n+3u\nbdy9PdAfeMXdB6102VPAEcn9Q5JrsrY/aMFXBSxZsCRbLy8iUuNV+ToFM7vEzPolD0cAG5rZVOB0\n4KxsvW/h0uXs1ekrDuj4IYu/V2IQESlLlSQFd3/V3fsm9y9w96eS+0vc/Q/uvpm793D3admKIb9+\nHY45+HtemLcd/Tp+xKL5i7P1ViIiNVZOrWg+8q7duOuoN3hl/jbs2/ETFs5VYhARKS2nkgLA4Xfu\nxr1D3mDM9104dsf30g5HRKRaqXEF8SrDgNt70bDxG3Tt2zbtUEREqpWcaykUO+CaXenQuy1Fy4u4\ncr83+H7GT2mHJCKSupxNCsU+eGAyFzyzA3tuNYP5Xy5MOxwRkVTlfFLYZlAXnvjb20xe2I4+neYw\n9wslBhHJXTmfFAD2vqIXT587lk9/as3unb/lm8+VGEQkNykpJP7vsl48e+E7zF7UhI+eydpyCRGR\nai0nZx+VZ/eLevHF4fNYr0NXAJYuKqR+w/yUoxIRqTpqKaxkvQ7NALjv+DfovOEsvpxUkHJEIiJV\nR0mhHB07FDJ3SWN67fATX7z/Q9rhiIhUCSWFcvT4ay9evm4SPyxrQK8ei5g6Xi0GEan9lBRWYfvT\nduOVGz5k0bI69Oq5lO9mqlaSiNRuGmiuQLehuzG6zuv896mlbNB6z7TDERHJKiWFDHQ54bd0OSHu\nL/xuGY02qJduQCIiWaLuo9Xw6rXjadNsMROe/DrtUEREskJJYTVs27c19VjGaUcWkL1NQ0VE0qOk\nsBqabNGSSw6YyJjvOvPY+RPTDkdEpNIpKaymo0f1pnO9Tzjzqg1Z+uOytMMREalUSgqrqU7Delx3\nQQHTfm7L8/+YkHY4IiKVSrOP1sD/nduDyVt8xNaH7JR2KCIilUothTW09SFbAzDng7kpRyIiUnmU\nFNbCw2ePp23XJky674O0QxERqRRKCmthj+M2p7Et5PQTFuOFRWmHIyKy1pQU1sIG7Rpz0cCpvPxD\nD54545W0wxERWWtKCmvpuDu7s2WD6Zxx86Ysm6cS2yJSsykprKW69fO49h/L+KqwNe+O1NiCiNRs\nmpJaCfY+ZXO+2GMerbrsknYoIiJrRS2FSmAGrbo0wx0+engyKowkIjWVkkIluuXYSXT94xZ8NOy1\ntEMREVkjSgqVqP/FW9EobxF/+SuwdGna4YiIrLasJQUza2Bm75jZ+2Y22cwuLuOawWY218zeS25H\nZyueqtCsVV0uOHo2zy3qzXPHPZl2OCIiq808S/3fZmbAuu6+0MzqAm8Ap7j726WuGQx0d/eTMn3d\n7t27+7hx4yo93sqybBl02mAWdRf9wPtfrEfddhunHZKICGY23t27V3Rd1loKHhYmD+smt1o/Aluv\nHlxzbR7zbUM+e+nLtMMREVktWR1TMLN8M3sP+BZ40d3HlnHZwWY2ycweMbNNshlPVek3pCWfz23C\n1kf1TDsUEZHVktWk4O6F7t4NaAP0MLPOK13yNNDe3bsCLwF3l/U6ZjbEzMaZ2bi5c6t/VVIzaLRB\nPX5eUsgbl7wChYVphyQikpEqmX3k7guAV4G9Vjo+392Lp+ncAWxfzvOHu3t3d+/evHnzrMZamS4Z\n/Dm7X/hbPv3Ho2mHIiKSkWzOPmpuZk2T++sAewIfr3RNq1IP+wFTshVPGk66viPr5C/jr5c2hgUL\n0g5HRKRC2WwptAJGm9kk4F1iTOEZM7vEzPol1wxNpqu+DwwFBmcxniq3UUvj3BMLeGrZ3rx81L/T\nDkdEpEJZm5KaLdV9SurKliyBrVrMo/GPs5g4qQ75XbZOOyQRyUGpT0mV0KABXH19PWjYkNnTFqcd\njojIKqlKahU4+M/rceAR65Gfn3YkIiKrppZCFTCD/HxYMGMh/xn8cPQpiYhUQ0oKVejckwo46O5+\nTDt3RNqhiIiUSUmhCp17a2vq5Dtn3rAxfP112uGIiPyKkkIV2nhjOGvoYh4tPJAxg0emHY6IyK8o\nKVSxMy5bnzbrFXDaK30penVM2uGIiPyCkkIVa9gQrry+Aa02zqOgwUZphyMi8guakpqCPw2uz4A/\nb5t2GCIiv6KWQgrM4r+f/28OD/a5Hb77Lt2AREQSSgopuviK+gwefThfnvrPtEMREQGUFFJ12S3r\nQ34+Z93bCSZNSjscERElhTS1bQt/Pa2QB+jPm4OHQw0rTigitY+SQsrOvHAdWjX5idMmHkbRQ4+k\nHY6I5DglhZQ1agRX3bgOPXaApTv1SjscEclx2k9BRCQHaD+FGuj1UV/yr62ugunT0w5FRHKUkkI1\nctMDzTnp4xOZceI/0g5FRHKUkkI1cuVNDVmeX59znt0FXnkl7XBEJAcpKVQjm24Kp58G93I47x4z\nHJYvTzskEckxSgrVzNnn16FFkyWcNu0kfOS/0g5HRHKMCuJVM+utB9fdXJ9vn2tA0cDD0LbOIlKV\nlBSqoYGDDAYlM8cKC2ODZxGRKqDuo2rKHe659Etua3khTJyYdjgikiMqTApmdpKZrV8VwUgJM3h8\nbCv+Ov8s5hx3keoiiUiVyKSl0BJ418weMrO9zIp3A5Bsu/r6eizNW4fz3tkPHngg7XBEJAdUmBTc\n/TygIzACGAx8ZmaXm9lvshxbzttsMxg61BjJkUw85S746ae0QxKRWi6jMQWPAklzkttyYH3gETO7\nKouxCXDeBXls2LSQ0+aejd99T9rhiEgtV+HsIzMbChwBzAPuBP7q7j+bWR7wGXBmdkPMbU2bwk3D\n6lJ/2kZwnKqoikh2ZTIltRlwkLt/WfqguxeZWd/shCWl9e8PsBUAPuVjbP2m0LJlqjGJSO2UyZjC\nBcCGZjbUzE42s+1KnZuS1ejkF6687GcO334y3m9/WLQo7XBEpBbKZErq+cDdwIZEq+FfZnZetgOT\nXyvKr8uoxQdz3rv7w+GHQ1FR2iGJSC2TyUDzAGAHd7/Q3S8EdgIGVvQkM2tgZu+Y2ftmNtnMLi7j\nmvpm9qCZTTWzsWbWfnU/QC456ywYMgQu5xzueHR9OPvstEMSkVomk6QwHWhQ6nF94PMMnrcU6OPu\n2wDdgL3MbKeVrjkK+N7dNwP+CVyZwevmLDO45RbYZx/neLuN566aBI8/nnZYIlKLZDLQvBSYbGYv\nAg78H/CGmd0I4O5Dy3pSMo11YfKwbnJbeVnu/sBFyf1HgJvNzLym7RFaherUgQcfNPrsnseM9qfC\n3pqRJCKVJ5Ok8HhyK/Zqpi9uZvnAeGAz4BZ3H7vSJa2BrwHcfbmZFRBjF/MyfY9c1KgRvPmWUafO\n7wHw777H5s+Djh1TjkxEaroKk4K7321m9YDNk0OfuPvPmby4uxcC3cysKfC4mXV29w9LXVJWyYxf\ntRLMbAgwBKBt27aZvHWtVyf55p59Fi7502z+26Q/Td99ETbaKN3ARKRGy2T2UW9ikdotwK3Ap2a2\n2+q8ibsvIFoYe610agawSfI+dYAmwHdlPH+4u3d39+7Nmzdfnbeu9Ro0gAmLtuSgmTeybL+DYfHi\ntEMSkRosk4Hma4HfuXsvd98N+D0xKLxKZtY8aSFgZusAewIfr3TZU8RqaYBDgFc0nrB6+vSBESPz\nGF3Um6PePRY/TFNVRWTNZZIU6rr7J8UP3P1TYtC4Iq2A0WY2CXgXeNHdnzGzS8ysX3LNCGJh3FTg\ndOCs1QtfAA47DC65BEZxGBc82hWu1CQuEVkzmQw0jzOzEcC9yeOBxODxKrn7JGDbMo5fUOr+EuAP\nmYUqq3LeeTD9C2fulP3wIzYqc7BGRKQimSSF44ETgaHEwPAYYmxBqhEzGH6HkZfXDTMoWracvKmf\nwtZbpx2aiNQgq+w+SqaUjnD369z9IHc/0N3/6e5Lqyg+WQ35+ZEcPvkEum38Le/teCx89FHaYYlI\nDbLKpJBMKW2eTEmVGqJRI/i+Xgv2XfQQX//+aPjmm7RDEpEaItMyF/8zs/PN7PTiW5bjkrXQujU8\n+3wdFq7TnH1n3k7BvgM0VVVEMpJJUpgFPJNc2zi5NcpmULL2unSBR5+ow5S8Thwy/ix+PuaEtEMS\nkRogk4Hmj9z94dIHzEwzhmqAPfeEO+7MY+TfO7Ho6KE0STsgEan2MmkplFWfWTWba4jBg2H0xxvT\npPe2sabt00/TDklEqrFyWwpmtjewD9C6uCJqYj1gebYDk8qTnw8FBbD/jrMZ/NmVDH5hAOyxR9ph\niUg1tKqWwixgHLCEWKxWfHuKKHUhNcg660Cdls05pug2Xtr/RpiinVRF5NesolJDZlY306qoVaF7\n9+4+bty4tMOokQoKYNcdl/HVp0t4o9Uf6TLxHmjRIu2wRKQKmNl4d+9e0XWZjCn0MLMXzexTM5tm\nZl+Y2bRKiFGqWJMm8OyL9WjUrAH7zL6TmXsfDYWFaYclItVIJrOPRgCnEV1H+g1Sw22yCfznhXoc\ntn9jfvzTkBhwEBFJZJIUCtz9uaxHIlWmWzd4/4sm5OX1xR2Kpn9F/qbavEhEMus+Gm1mV5tZTzPb\nrviW9cgkq/LywB1OPHgOx272Ej7yX2mHJCLVQCYthR2T/5YeoHCgT+WHI1XJDJpt1ZxLHz+S9sdc\nwHmbjobdd087LBFJUSZ7NOu3RC128WX5TP98Gec/eAnt+x7LoPGtYMst0w5LRFKSyR7NG5nZCDN7\nLnm8tZkdlf3QpCqYwZ331GP3nos5ctFNvLL7pfD992mHJSIpyWRM4S7geWDj5PGnwKnZCkiqXr16\n8Niz67Btp2Us2m2vmLsqIjkpk6TQzN0fAooA3H05mppa6zRtCm9NakTfBw+DvDyWz/yGKJYkIrkk\nk6Twk5ltSAwuY2Y7AQVZjUpSkZf8NIy4dgE7t5/FwrP/nm5AIlLlMkkKpxP1jn5jZv8D7gFOzmpU\nkqqNt2rChMKuHHrVdiwfcXfa4YhIFaowKbj7BKAXsDNwLNDJ3SdlOzBJz977GLfe4jzLvpw4ZBk+\n+tW0QxKRKpJJSwF3X+7uk939w+pUHE+yZ8jxdTjn9CUMLzqGK/d5TfswiOSITBavSY667JoGfPn5\nQoq+6gStWqUdjohUASUFKZcZ3Pt4I8wOAeDn7xdS15bHVCURqZUyWby2i5mtm9wfZGbXmVm77Icm\n1YFZ/Pet15ezZasFjGl/ONx2GyzX5nsitVEmYwrDgEVmtg1wJvAlMQNJckirTeqQ16IZuxc8zmXH\nz6Bw2+7wyitphyUilSyTpLDcY3u2/YEb3P0GoHF2w5Lqpn17mDC5Af0H5HE+l/H7qTczZ48BMGpU\n2qGJSCXKJCn8aGZnA4OA/5hZPlA3u2FJddS4MYwaZYwYAW/aLtz6f0/AAQfEyQ8/jP0+RaRGyyQp\nHAosBY5y9zlAa+DqrEYl1ZYZHHkkTJxonPfMTtCoEdOmFrH8kP7QsSPccYe2+BSpwTJqKRDdRq+b\n2eZAN+D+7IYl1d0WW0QhvZ9+gt598ujd4G2+brcrDBkC228Pr76adogisgYySQpjgPpm1hp4Gfgz\nUTl1lcxsEzMbbWZTzGyymZ1SxjW9zazAzN5Lbhes7geQdK27LlxxBbz/eSO6TXuUp88cE6W3d98d\nXnwx7fBEZDVlkhTM3RcBBwE3ufuBQKcMnrccOMPdtwJ2Ak40s63LuO51d++W3C7JOHKpNgYMgAkT\noF07o99Vv+X0/T/n55tvhz7J5nxvvgk//phukCKSkYySgpn1BAYC/0mO5Vf0JHefndRNwt1/BKYQ\n4xFSC3XsCG+9BSefDB9+XIf844dAfj4sWgT9+sUFI0eqHLdINZdJUjgVOBt43N0nm1kHYPTqvImZ\ntQe2BcaWcbqnmb1vZs+ZWSYtEKmm6teHG2+EZ56JMtwzZ8KjzzWEZ5+FDh3gqKNghx3g9dfTDlVE\nypFJldTX3L0fcKuZNXL3ae4+NNM3MLNGwKPAqe7+w0qnJwDt3H0b4CbgiXJeY4iZjTOzcXPnzs30\nrSUl9erFf6+6Cg45BI7/Vw8Wv/Q/uO8++PZb2G23mMIqItVOJmUuupjZROBD4CMzG5/pX/RmVpdI\nCPe5+2Mrn3f3H9x9YXL/WaCumTUr47rh7t7d3bs3b948k7eWauCaa+DMM6Mqxk49jY+3GwCffAL3\n3w+dO8dFzz0HCxemG6iIrJBJ99HtwOnu3s7d2wJnAHdU9CQzM2AEMMXdryvnmpbJdZhZjySe+ZkG\nL9Vb3bpw5ZXRezRrFnTvDq++0xD6948L5syB/feP+a333KPxBpFqIJOksK67rxhDcPdXgXUzeN4u\nwGFAn1JTTvcxs+PM7LjkmkOAD83sfeBGoH9SUkNqkb33hvfeg4MOgm7dSp1o2RJeew1at4YjjoCe\nPWO0WkRSYxX9Djazx4m+/3uTQ4OA7u5+QJZjK1P37t193Lhxaby1VJKlS2HwYDj7bOjalWghjBoF\nZ50F8+bBV19FwhCRSmNm4929e0XXZdJSOBJoDjyW3JoRC9hE1si0abHgeccdYfhwcMuDww+P3d2e\nfLIkIfz737FkWkSqzCqTQlL87hx3H+ru2yW3U939+yqKT2qhrbaC99+PSUjHHhtDDAUFQKNG0dcE\nMTtp4MAYb7jvPlCvokiVWGVScPdCYPsqikVySIsWMfHoiivg0UdjSOEXOneO9QwtW8KgQbDzzjC2\nrGUuIlKZMhlTuBboCDwMrGjLlzXFtCpoTKH2efNNWH/9aEEsXRrrHIp3fKOoKGYmnX127PY2c2bJ\nQggRyVhljilsQEwT7QPsl9z6rl14IiV23jkSgnssej7gAPjuu+RkXl6MSn/6KTzySCSEwkL4wx/g\nwQe1LahIJatT0QXurkFlqTI9esBf/hJTV++/H3bZJTnRuDH06hX3Z8yASZMiSbRvD6edFps8NGqU\nVtgitUYmK5rvNrOmpR6vb2YjsxuW5CIzGDo0upPq1o0ccMUVZaxpa9cOpkyBJ56INQ6nnAJt28JH\nH6USt0htkkn3UVd3X1D8IJl5tG32QpJc1717lOI+5JConzRnThkX5eXFaug33ogs0r9/zFQCePrp\nSBoistoySQp5ZrZ+8QMz24AMup1E1kaTJtF9NHEibLxxtBYmTCjn4p494dZbo1R3URGcdBJsvTXs\ntx+MGaPprCKrIZOkcC3wppldamaXAG8CV2U3LJHoTmrXLu6PGBEtiEMPjURRrrw8GDcOLroI3n47\n+qB23FHlukUylEnp7HuAg4FvgLnAQe5+76qfJVK5BgyAv/0t1jZst12scSu3EdC8OVx4YZTLGDYs\ntgctvnD+fK2SFlmFCtcpVDdap5DbFiyInqLrr49upYkTS61pKE9hYbQgzKJr6d//hhNOiG3iNtqo\nSuIWSVtlrlMQqTaaNoVzzoHp0+Ghh+L3/IIFsOuuUQ2jzGUL+fklmWPgQOjdGy6/PPqmhgyBjz+u\nwk8gUr0pKUiN1LAhbL553P/66+ghGjQojg0bBkuWlPPEnj3hscciEfz5z3DvvXD11VUWt0h1p6Qg\nNV6XLvDBB7FsoUWL6Blq3z6GD8pVnD2+/BIuvTSOvfNODEo/8kh0OYnkICUFqRWKly289RaMHg3H\nHAMbbhjnHnkktoYuU4sWMTgB0Q81f36U0Nhiixi8WLSoSuIXqS400Cy12vz58Ts/Lw+OPhrOOCNa\nEeUqLIw9Ha6+Oqa0brZZ7Cudp7+fpGbTQLMI0VqYNCmmtN5+e/yOP/zwmK1apvz82Df0zTdjtfRF\nF0VCKCqC88+PFXQ17A8pkdWhloLkjBkz4Lrr4F//ik1+2raN3qGGDTN48gcfwA47RG3vLl1iA4iB\nA7VtqNQYaimIrKRNm0gKs2ZFQoAo07377vDCCxU0ALp0iSfeemtkkb/8JV5QG/9ILaOkIDlnnXXi\nv+6w116xVcPvfx8NgVVOPNpgAzj++BhrmDIlupO22y7OXXUVHHdcjHTXsNa3SGlKCpKzzOD002Ha\nNLjjDvjhh5h4dNttGTx5yy2jlEbduvF43rzYIW7nnePc5ZfHAgqRGkZJQXJe/foxM2nKlNjMbdCg\nOP7UU/DPf8LChRm8SHGN7xEjYpzh3HNjqlOxclfTiVQvSgoiifx8+OMfo2w3wH/+Ey2Jdu3g4otL\nbRFanvXWix3gXnsNPv+8ZFHcJ59Ekb7ic7/aNUik+lBSECnH7bfD//4XW4JedFEMTl9+ecn5VQ4d\ndOhQsulPcbZ55JGou/Sb30TXU4VZRqTqKSmIrMLOO0c30qRJ0a20ySZx/JtvYifQgQOjx2j69FW8\nyGabxUVz5sCoUdCxI1xzTcmCuClToKAg2x9FJCNapyCyBqZNi8lHL78cCQKicXDXXfDb30YrYpUl\nvRcsiJKvEPWWJk2KRXNHHAF77BGtC5FKpHUKIlnUoUOU6p49Gz78EG64IZYytGkT50eOhK5d4dRT\nY8voH35Y6QWKEwLAjTdGxdbnnou5se3axXQokRQoKYisBTPo1AmGDo0qrZtuGsebN49ae7ffDv36\nxRKHnXcumYT0iwb6jjvGorjZs+Hhh2HbbUtaCt99F+c0/iBVRN1HIlm0ZEmsZ3v55Rh3GDUqjv/x\nj/F7fo894rb99uX0GI0aBYcdBvXqRXY57jjo0yeD7eZEfinT7iMlBZEUnH9+FGP94IN43KRJ/L6/\n4oqVLnSPQk133x0JYt68aJq8/TY0alTlcUvNpTEFkWrs0ktjbPmbb+CBB6Ll0KpVnFu8OLqhBgyA\nESON6U27xSq6r7+OkezevUsSwn33VTD1SWT1ZK2lYGabAPcALYEiYLi737DSNQbcAOwDLAIGu/uE\nVb2uWgpS282ZE4uhX3kl7kMMbN90E+yzT6kLCwoikyxdGjsMDR0KvXqpa0nKVB1aCsuBM9x9K2An\n4EQz23qla/YGOia3IcCwLMYjUiO0bBkNgFmzYPLkmJzUpQtstFGc/+STuNGkSVTz+9vfYMyYKPe6\nzTaxrajIGspaUnD32cV/9bv7j8AUoPVKl+0P3OPhbaCpmbXKVkwiNYkZbL01nHxyzGzafvs4fs45\ncfyoo+DLwjYlxfdGjIgCfcV7PEyZoqJ8stqqZEzBzNoD2wIrF59vDZT+qZ3BrxMHZjbEzMaZ2bi5\nc+dmK0yRGmHYMDjllGhNbL553P/mh3WittL48SWbRZxxRgxO/OEP8PrrKuktGcl6UjCzRsCjwKnu\nvvISnrI6P3/1k+vuw929u7t3b968eTbCFKkxWrSIzYI++ywWQN9ySyxl+JVhwyIxvPwy7LZbNDUe\ne6zK45WaJatJwczqEgnhPncv66dxBrBJqcdtgFnZjEmktthkExg+PHqJTj89jj33HPz970m573bt\n4MorYx/S4cPh558jkwAsWwYzZ6YWu1RfWUsKycyiEcAUd7+unMueAg63sBNQ4O6zsxWTSG3UsWNJ\nue+XXoLzzotCrDfeGBOTaNgQjjkm5sAWZ49HHoH27aF/f3jzTXUtyQrZbCnsAhwG9DGz95LbPmZ2\nnJkdl1zzLDANmArcAZyQxXhEar1rr40V1J06xVhDx45ROQOIkevineJ22SUu+O9/4/4OO8TOceXu\nRSq5QiuaRWqpl1+OmUrHHBM7yy1fHtW680r/KbhwYayUvvHGOPHBB5E8Fi2KFobUGtVhnYKIpGiP\nPaIaxpFHxuNhw2C77WJHuRV/CzZqFPU1Jk+GF1+MhPDjjzEeMXAgjF15wqDUdkoKIrWYWUnLoE2b\n+H3fty/sumvsDPqLC4vrbCxbFjsKPfMM7LRT3P797zgutZ6SgkiOOPBA+PhjuO22KJfUu3fMWP2V\nDTeMWkszZkRtje+/j1bDhx/G+SlToptJ4w+1ksYURHLQ4sWxtmH77SM5zJsH334bK6V/pagoRq93\n2SUeH3FEDEo3bhx7QfTsGZtF7LVXVX4EWU0qnS0iGfvb32Lb6EGD4KKLSjYLKtMXX8Abb0SieOut\nmOq6+ebRggC4/voYpN5pp5gGpa1FqwUlBRHJ2Pz5sc7tppuiV+iYY2K9Q6tMKpEtXBgL4bbYIh5v\nuWVSsY9oTfToEXXAi0e8JRWafSQiGdtwQ7jqKvj885i+Onx4OeMNZWnUqCQhQLQYPvssupgGDYqM\nM3VqnFuyBDp3hsGDY6/SSZM0NlHNqKUgIr8ybVrMWmrfPganH34YTj01/vBfI+4xw2nWrJgC+9Zb\nMZAB8aJ33AGHHhqDHYsWRZaSSpVpS6FOVQQjIjVLhw4l9//zH7jggljfduCBsY9Pr14xxTVjxRv/\nbLwxPPVUJIlp00rGJTbfPM6/8AIccEA87tkzbjvtFK0LjU1UCbUURKRC774b+0e//HJs+NayZfzR\nbxZVudu0iVbFWm/6NnVq1GUqThbFpfKnTImxio8/jqXZnTpph7nVpIFmEal0hYUxDDB7dsnWoO3a\nwVdfRdXW3XaLVsQee/yytbHUNBVwAAANzElEQVRG3GOm09ix0bWUlxdjEXffHRlov/1iJV6vXlC/\n/lq+We2npCAiWeceFTJee63k9u23cOyxsUiuqAjuvDNWUG+1VSX8cT9zZvRnPf10lIRdsgS6doX3\n34/zqtlULiUFEaly7jEbtU4d2GyzWPjctWuca9aspCVx0EGrOSZRlkWL4JVX4KefoiVRWAitW0cT\npW/faEl07qxupoSmpIpIlTOLrv/NNovHnTvHMMHIkbDvvjBhQlTsnjQpzn/4YewiN378GsxMbdgw\nfvkfemg8XrIEjj8+ajSde25ko003jTEKyZhaCiJSpb76Cpo3h3XWiRlNp5wSx9dbLypp9OoFJ54Y\nyx/W2KxZJd1Mp50Gu+8O77wTK/T69o0M1aJFpXyemkLdRyJSI8ycCWPGlIxJfPVV1OCrVy/GI+bM\niUTRo8dajic/8QScdFK8oVnUbdpvPzj55LVYgFFzKCmISI1UUFCyvejAgVG1G6BBg5jV1L9/LJRe\nI+7w3nvRgnj6afj005j2Wq9erJ+oVy8qBDZoUBkfpVpRUhCRWmH+/FgLMXp0/B7fbruSYYI774zB\n6+K1b6vthx+i3wqiKfLuu7DuuvC735V0M220UaV8jrQpKYhIreMe9fcaN45eoOIZTFtuGQuhDzgg\ntpvOW5MpNIsXR+Z55pnIPjNmwMEHl2Sg4gV0NXQ2k5KCiNR6X30FTz4Zt1dfjRlM994b3UuLFkVl\njDUah3CPtQ9msM02sYiuQ4doNfTuHQPXvXtHE6WGJAlNSRWRWq9t2xgnfumlWDR3772w995xbuTI\nmOV06KFw//0xVpExM+jWLRICwAYbwIgRsOee0Zd13HHRaihuRcybFyVma9gf2WVRS0FEaqW3347E\n8NRT8M03ULcu9OkTvUN11qYUqHssvhg9OvqrWrQomVu7ySbRiihuSbRvX0mfZu2p+0hEhCi1MXZs\ndDHNnBmtCYhS4M2axe/1ta6vN306PPts9GG9+mrMaMrLi7m1660Xq/SaNq2EZdxrTklBRKQcRUXx\nx/yYMfH4N7+B/fePsYhtt62EF//oo6jx8ac/xbE994wSs5tt9ssxiY03Xss3y5zGFEREypGXFwvl\nZs6Mwn0dO8LNN8N//xvnFy6MbqfFi9fwxTt3LkkIELU8rrsuqgI+/HAswDjssJLzL7wQfVzVgFoK\nIiLEkoWioujlefRROOSQKK/0u99FF1PfvpW0IVxhYSygW7YsNhEqKIiB7KIi2HrrklbE7rtX6g50\n6j4SEVlDy5ZFS+LJJ6M6xsyZ0QCYPj3Gkj/7LB536FAJM1ILC2HcuBiLGD0a3ngjKr9ef30MXhev\n3uvVC9Zff43fRklBRKQSuEd11+efh7PPjiRQvNdPixaxW2jPnrDzzrG6eq39/HMkiXbtYszh/vth\nwIBYE/HJJ2v8skoKIiJZMmVKtCSKdw397LNYtjBlSpwfNixWXffsWQmtiaVLo8JrQUH0Ya0hJQUR\nkSoyf350MRVvKLTpptHVBLGArmfPWEQ3YEBqIWacFNZmCYeIiBDjwaXHhKdOjW1Ki1sSb70VVTMG\nDIi9gHr1gu7dI1lUSmuiEmWtpWBmI4G+wLfu3rmM872BJ4EvkkOPufslFb2uWgoiUhMVFcXg9IwZ\nMSYxdmxMfYVoTQwbFvX3li6F5cujWGtlqg4thbuAm4F7VnHN6+6+5p1kIiI1RHHl1jZtolZTYWG0\nJt5+O1oS7drF+eefjz2su3YtaUlUZWsia0nB3ceYWftsvb6ISE2Wnx+/+Lt2hSFDSo537BiznN56\nC+65B269NY5/9FGsfcu2tMcUeprZ+8As4C/uPjnleEREUrXVVnDppXG/sDCSwdixsMUWVfP+aSaF\nCUA7d19oZvsATwAdy7rQzIYAQwDatm1bdRGKiKQoPx+6dIlbVUmt9pG7/+DuC5P7zwJ1zaxZOdcO\nd/fu7t69efPmVRqniEguSS0pmFlLsxg2MbMeSSzz04pHRESy2H1kZvcDvYFmZjYDuBCoC+DutwGH\nAMeb2XJgMdDfa9pKOhGRWiabs4/+VMH5m4kpqyIiUk1oPwUREVlBSUFERFZQUhARkRWUFEREZIUa\nVzrbzOYCX67h05sB8yoxnJom1z8/6N9Anz93P387d69woVeNSwprw8zGZVIlsLbK9c8P+jfQ58/t\nz58JdR+JiMgKSgoiIrJCriWF4WkHkLJc//ygfwN9flmlnBpTEBGRVcu1loKIiKxCziQFM9vLzD4x\ns6lmdlba8WSbmW1iZqPNbIqZTTazU5LjG5jZi2b2WfLf9dOONZvMLN/MJprZM8njTc1sbPL5HzSz\nemnHmC1m1tTMHjGzj5Ofg5659P2b2WnJz/6HZna/mTXIpe9/TeVEUjCzfOAWYG9ga+BPZrZ1ulFl\n3XLgDHffCtgJODH5zGcBL7t7R+Dl5HFtdgowpdTjK4F/Jp//e+CoVKKqGjcA/3X3LYFtiH+HnPj+\nzaw1MBTo7u6dgXygP7n1/a+RnEgKQA9gqrtPc/dlwAPA/inHlFXuPtvdJyT3fyR+IbQmPvfdyWV3\nAwekE2H2mVkbYF/gzuSxAX2AR5JLau3nN7P1gN2AEQDuvszdF5BD3z9RBXodM6sDNARmkyPf/9rI\nlaTQGvi61OMZybGcYGbtgW2BscBG7j4bInEALdKLLOuuB84EipLHGwIL3H158rg2/xx0AOYC/0q6\nz+40s3XJke/f3WcC1wBfEcmgABhP7nz/ayxXkoKVcSwnpl2ZWSPgUeBUd/8h7Xiqipn1Bb519/Gl\nD5dxaW39OagDbAcMc/dtgZ+opV1FZUnGSvYHNgU2BtYluo9XVlu//zWWK0lhBrBJqcdtgFkpxVJl\nzKwukRDuc/fHksPfmFmr5Hwr4Nu04suyXYB+Zjad6C7sQ7QcmibdCVC7fw5mADPcfWzy+BEiSeTK\n978n8IW7z3X3n4HHgJ3Jne9/jeVKUngX6JjMPKhHDDg9lXJMWZX0n48Aprj7daVOPQUckdw/Aniy\nqmOrCu5+tru3cff2xPf9irsPBEYTW8FC7f78c4CvzWyL5NAewEfkyPdPdBvtZGYNk/8Xij9/Tnz/\nayNnFq+Z2T7EX4r5wEh3/3vKIWWVme0KvA58QEmf+jnEuMJDQFvif5w/uPt3qQRZRcysN/AXd+9r\nZh2IlsMGwERgkLsvTTO+bDGzbsQgez1gGvBn4g/BnPj+zexi4FBiJt5E4GhiDCEnvv81lTNJQURE\nKpYr3UciIpIBJQUREVlBSUFERFZQUhARkRWUFEREZAUlBamxzOxVM8v6frtmNjSpMnrfGj7/EjPb\ns4Jr+pVXvdfMFq7m+x1QUcFHM+tdXDlWpLQ6FV8iUvuYWZ1SNXAqcgKwt7t/sSbv5e4XZHDNU1Te\ngsoDgGeIxVoiq0UtBckqM2uf/JV9R1Lb/gUzWyc5t+IvfTNrlpSkwMwGm9kTZva0mX1hZieZ2elJ\nYbe3zWyDUm8xyMzeTGrm90iev66ZjTSzd5Pn7F/qdR82s6eBF8qI9fTkdT40s1OTY7cRxeWeMrPT\nVro+ozjN7C4zOyS5P93MLjazCWb2gZltWeq1bl7Fv+O1yXNeNrPmybFjks/4vpk9mqze3RnoB1xt\nZu+Z2W/MbDMzeym5boKZ/SZ52UZWst/CfcnKX8xsezN7zczGm9nzpcpiDDWzj8xskpk9kOnPgNQw\n7q6bblm7Ae2JFaXdkscPEatIAV4l6t0DNAOmJ/cHA1OBxkBzosLlccm5fxLF/Yqff0dyfzfgw+T+\n5aXeoynwKVEQbTBRE2iDMuLcnlj9vS7QCJgMbJucmw40K+M5mcZ5F3BIqdc6Obl/AnBnqde6uZx/\nQwcGJvcvKL4O2LDUNZeVet0V75c8HgscmNxvQJSR7p3E24b44/AtYFegLvAm0Dy5/lCiAgBEnaD6\nxf+uaf9s6Zadm7qPpCp84e7vJffHE4miIqM99oH40cwKgKeT4x8AXUtddz+Au48xs/XMrCnwO6IY\n3l+SaxoQZR0AXvSyyzrsCjzu7j8BmNljwG+JUgiVEWdpxcUJxwMHVfD6EGVKHkzujyr1/M5mdhmR\n+BoBz6/8RDNrDLR298cB3H1JchzgHXefkTx+j/heFgCdgReTa/KJ0tMAk4D7zOwJ4IkM4pYaSElB\nqkLp2jKFwDrJ/eWUdGE2WMVziko9LuKXP7cr12lxokT2we7+SekTZrYjUUK6LGWV1c5EpnGW9ZzC\nla+x2CWwuNz3U172eETxZ74LOMDd3zezwcRf/ytb1eda+Xupk1w/2d17lnH9vkSLrB9wvpl18szH\nZaSG0JiCpGk60W0DJZUrV9ehsKIAYIG7FxB/MZ9cqo982wxeZwxwQNIvvy5wIFFQsEq5e6G7d0tu\nxQkhj5J/nwHAG8n9xsBsixLpA0u9zI/JOTz20JhhZgcAmFl9M2u4ihA+AZqbWc/k+rpm1snM8oBN\n3H00sXFRcetEahm1FCRN1wAPmdlhwCtr+Brfm9mbwHrAkcmxS4mKuJOSxDAd6LuqF3H3CWZ2F/BO\ncuhOd6+o66iq/AR0MrPxxDjAocnx84nxgi+J7qrGyfEHgDvMbCiRTA4DbjezS4CfgT+U90buviwZ\nFL/RzJoQvyOuJ8ZlRiXHjNjneEHlfkypDlQlVUREVlD3kYiIrKCkICIiKygpiIjICkoKIiKygpKC\niIisoKQgIiIrKCmIiMgKSgoiIrLC/wN4lEbSbKp0pAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266abca0940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We want to lot the data \n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Define weights of zipfian distributuion\n",
    "def zipf(index):\n",
    "    return 1.0 / (index + 5)\n",
    "\n",
    "# Use zipifian distribution for the classes\n",
    "def zipf_sampling_weights():\n",
    "    return np.asarray([ zipf(i) for i in range(Param.num_classes)], dtype=np.float32)\n",
    "\n",
    "data_sampling_distribution = lambda: zipf_sampling_weights() / np.sum(zipf_sampling_weights())\n",
    "\n",
    "print(\"start...\")\n",
    "\n",
    "\n",
    "# Train using uniform sampling (like before)\n",
    "np.random.seed(1)\n",
    "softmax_sampling_weights = lambda: np.repeat(1.0/Param.num_classes, Param.num_classes)\n",
    "minibatch_data, cross_entropy_data, _ = train(do_print_progress = False)\n",
    "\n",
    "# Train using importance sampling\n",
    "np.random.seed(1)\n",
    "softmax_sampling_weights = zipf_sampling_weights\n",
    "minibatch_data2, cross_entropy_data2, _ = train(do_print_progress = False)\n",
    "\n",
    "plt.plot(minibatch_data, cross_entropy_data, 'r--',minibatch_data, cross_entropy_data2, 'b--')\n",
    "plt.xlabel('number of mini-batches')\n",
    "plt.ylabel('cross entropy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In the example above we compare uniform sampling (red) vs sampling with the same distribution the classes have (blue).\n",
    "You will need to experiment to find the best settings for all the softmax parameters.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## What speedups to expect?\n",
    "\n",
    "The speed difference between full softmax and sampled softmax in terms of training instances depends strongly on the concrete settings, namely\n",
    "* Number of classes. Typically the speed-up will increase the more output classes you have.\n",
    "* Number of samples used in sampled softmax\n",
    "* Dimension of hiddlen layer input\n",
    "* Minibatch size\n",
    "* Hardware\n",
    "\n",
    "Also you need to test how much you can reduce sample size without degradation of the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start...\n",
      "Measuring speed of sampled softmax for sample size 5 ...\n",
      "Measuring speed of sampled softmax for sample size 10 ...\n",
      "Measuring speed of sampled softmax for sample size 100 ...\n",
      "Measuring speed of sampled softmax for sample size 1000 ...\n",
      "Measuring speed of full softmax ...\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYVOXZx/Hvj46AIoqEpliwGxGx\n997FxBYbiChK7DVqbK8aS9TYoiCWgD1YItgLYCyxgWKJFQWlRUABCzbgfv+4n5Fhmd09wM7Olvtz\nXXPtaXPmPmX2mfNUmRkhhBBCWQ1KHUAIIYSaKRKIEEIIBUUCEUIIoaBIIEIIIRQUCUQIIYSCIoEI\nIYRQUCQQRSLpeUlHlzqOrCSZpDWq+70V7PN3kiZK+k7SRlW576omaVdJj+TNl3s+JB0p6aUq+Mym\nkj6UtNLS7msp48h8nSRtL2lS3vwESTtXcTxbSfokxbNfVe67qklaV9LoKtxf2fP7uqT1lmafdS6B\nkLS1pP9Imi3pa0kvS9qk1HHlk3SRpItKHUcNdzVwgpm1NLO3ivHPJKuU2G9fwSaXAVdUUzgAmNlP\nwB3An6rzc1MCNzhv0ULXqTpjKcfFwN9TPI+U8oeapMGSjqxgk0vw81csV+PnY4nVqQRC0rLAY8CN\nQBugI/B/wE+ljCsskVWA/5Y6iMqkHx/LmdmrJfj4e4HekpqW4LNzatp1qmnxFCSpPbAD8Eg56xtV\nwccMB3ZIn7VE6lQCAawJYGb3mdk8M/vBzJ4xs3fg118/L0u6MT1hfChpp9ybJS0n6XZJUyVNlnSp\npIZ564+S9IGkmZKelrRK3rpd0v5mS/o7oCwBS1pR0mOSZqUnnhclNUjrzpb0qaRvJb0v6Xd578sd\ny7XpvZ9J2jItnyhpmqTeedsPljRQ0rNpf//Oj79MTE0lXS3pC0lfpvc1z1t/ZjpHUyQdVcnxHZli\n+1bSeEmHpeUNJJ0n6fMU653p/DeV9B3QEHg7Hf9dwMrAoynr4CxJXVJWTp90vDMlHSdpE0nvpHPy\n97w4Vpc0UtJXkmZIukdS67x1X0vqnuY7pG22z3AJ9wD+XWD5num4Z0i6KndNy5yb3DE0ylu20C/e\niu45M5sEzAQ2L7DvDpJ+kNQmb9lGKZ7GktZI98DstOyfGY41f/+LXKe0fKHstXTfXbo4+07v2zPd\n89+m7+IZeeuOkTQuXbPhkjqk5Z8Cq7HgPrkc2Ab4e5r/e16Mf5RnRX0r6ZJ0D7wi6RtJQyU1Sdsu\nL/9+Tk/X4DFJndK6NpImSdonzbdMcfXKcIi7AG+a2Y95xzVB0p8kvQN8L6lRuo4Ppc8fL+mkvO2b\np/M7U9L7wEI5JWnfY4BdF/f85++kzryAZYGvgCH4F3f5MuuPBOYCpwKNgYOB2UCbtP4R4BagBbAS\n8DpwbFq3HzAOWAdoBJwH/CetWxH4Bjgg7ffU9DlHZ4j5cmBgel9j/IZWWncg0AFPyA8GvgfalzmW\nPviX9FLgC+AmoGm6Kb4FWqbtB6f5bdP664GX8uIwYI00fR3+66MN0Ap4FLg8rdsd+BJYP52ne/Pf\nW+bYWqTzslaabw+sl6aPSudzNaAl8DBwV6F40vwEYOe8+S5pm4FAs3S8P6ZruBL+9DgN2C5tvwb+\npWwKtAVeAK7L298xwAfAMsDTwNUZ77kHgDPLLDNgVDp/KwMf5+6FdN1eKnMMjfLe+3zetuXec3nb\nDwdOKie2kcAxefNXAQPT9H3An/F7qxmw9RJ+58pep7Lzg4FL0/T2wKTyrmmZ/U4FtknTywPd0/SO\nwAyge7qWNwIvVHCf/Ho+y8Q4HP9/sR6ewzAi3YvLAe8DvdO2KwD7p/uiVbrej+Tta1fgf+meuxV4\nMON5uwq4qcyyCcBYoDPQPF2bMcAFQJMU32fAbmn7K4AX033WGXgv//ymbW4A/rYk19bM6lYCkU7I\nOummnIT/Ax0OtEvrjgSmkP4Bp2WvA0cA7dKN0jxv3SHAqDT9JNA3b10DYA7+SNsLeDVvndLnZ0kg\nLgaGUeAfbIFtxwI9847lk7x1G6Qbv13esq+Abml6MHB/3rqWwDygc96XZo0U+/fA6nnbbgGMT9N3\nAFfkrVuTihOIWekL1rzMuhHAH/Pm1wJ+If2zLLtPyk8gOpY53oPz5h8CTinnXO4HvFVm2XDgXeAd\noGnG++1Z4LgyywzYPW/+j8CIvOuWNYEo957LW3YPcEE5sR0NjMy7JycC26b5O4FBQKel/L4VK4H4\nAjgWWLbM8tuBv5a5j38BupRzn/x6PsvEuFXe/BjgT3nz15D346HMe7sBM8ssuzHdN1OAFTKet1vJ\n+x7lxX5U3vxmwBdltjkH+Eea/qzMfdaPRROIvwB3LOn1rWtZTJjZB2Z2pJl1wn/ldsB/EedMtnTm\nks/TNqvgv+CnpuyJWfjTRK6WyCrA9Xnrvsa/dB3T+yfmxWD585W4Cv+V+EzKkjg7t0JSL0lj8z5z\nffxpJefLvOkf0meXXdYybz4/xu/SMXQoE09b/NfSmLzPfSotp+yx4uevIDP7Hn/yOQ4/r49LWjtv\nP/nv/Rz/ldyuvP2Vo+zxFjx+SStJuj9lV3wD3M3C5xL8S7s+cKN5IXAWM/FflmWVPUdlz3MWFd1z\nOa3wRLiQB4EtUhbMtvg/xhfTurPSvl6X9F9VklVYAvsDewKfp6ywLdLyhe6bdB9/xcLnJIus980y\nkm6RZ4V+gz95tlZe1jOe0K6P/+P+KuPnZ7lvVgE65K5/ugfOZcF3JMt3saL7o1J1LoHIZ2Yf4r9g\n1s9b3FFSfvnAynjKPxF/gljRzFqn17JmlqsmNhHPbmqd92puZv/BH4c753aY9t+ZDMzsWzM73cxW\nA/YBTpO0U8prvhU4Af9V0hp/hMxUtlGO/Bhb4o+mU8psMwP/gqyXd5zLmVkuoVnoWPHzV9HxPW1m\nu+DZSx+mYyJ97ipl9jOXhb+oC+2qos/J4PK0j9+a2bLA4eSdy3Q+rsN/oV6Un3dfiXdIZV9llD1H\nZc8z+JMaeIKc85u86YruuZx1gLcLBWZms4BngIOAQ4H7cj+OzOx/ZnaMmXXAf6nfrKqpqjynguPJ\nzMzeMLOe+A+0R4ChadVC942kFng20OTydrUkn5/ndPzpdrN032yb++j0+Q3xH5J3Av0X4xyWd9/k\nxzsRf3LPv/6tzGzPtD7Ld7Hc+yOLOpVASFpb0ul5hUid8Wyi/BomKwEnpYK6A/ET+ISZTcW/TNdI\nWlZeiLq6pO3S+wYC5yjVK5YXqB6Y1j0OrCfp9/ICx5PI+MWQtHcqMBSeXz8vvVrgN8v0tF0fFk7o\nlsSe8mrATfAqdq+Z2UJPOmY2H/8nfq1SHXtJHSXtljYZChwpr8O9DHBhBcfWTtK+6Uv8E/BdOjbw\nPPBTJa2a/jlfBvzTzOaWs7sv8TzYJdUqff4sSR2BM8usvx4YY2ZH49dzYMb9PgFsV2D5mamAszNw\nMrBIIbCZTcf/sR0uqWH6Fb963iYV3XOk42jDwvd3WffiWaD7p+ncew/MfU/wX7PGgmuzNMYCh6bj\n2Z3C56ZCkppIOkzScmb2Cwu+F+DH0EdSN3ntrcvw+3hCOburivvmB/y+acOi9/u56e9ReLXSO8s8\nXZTnWaC7pGYVbPM68E0quG6ezun6WlBtfyh+fyyfruWJ+W9O52fj9FlLpE4lEHgh7GbAa5K+x784\n7+G/AnJeA7riv5T/AhyQ91jYCy8Meh//0jyI//LFzP4FXAncnx4138MLwjGzGXiB8hX4425X4OWM\nMXcFnsP/eb0C3Gxmz5vZ+3he6Cv4Tb7BYuyzPPfiN/jX+I1zWDnb/QnP9no1Hetz+K8ozOxJ/Jf2\nyLTNyAo+rwF+7qekz9wOz48HL8u4C39kH48XMJ9YYB85lwPnpUftMyrYrjz/hxdszsYTgIdzKyT1\nxAvfj0uLTsO/vOWdn1+Z2ZvAbEmblVk1DM/bHps+7/ZydnEMnlh9hReY/vp0UNE9lxwKDKkkO2w4\nfo99aWb5vyQ3wb8n36VtTjaz8QApy6nSYy/HyfiT8Cz8/ipYjTODI4AJ6biPw5/4MLMRwPl4+dJU\nPEH9QwX7uR44INX0uWEJ4rgOLzCegf8/eSq3QtLG+L3Sy8zm4dfKgLML7GchKSt4JNCzgm3m4eey\nG/4dmQHchhekg9/Tn6d1z+Dfp3z7As+bWaGn10xytWXqBXmjlaPNbOtSx1Ld5I2bJpnZeaWOpa6R\ntCte4F5tLXfTr8O38ULnadX1uaHqSFoXr3G5qRXhH7Gk1/BKDu8t6T6qojFGCPWamT2D/4Krzs/8\nCVi70g1DjZVyCYrWy4OZlX2qXWx1LYsphBBCFalXWUwhhBCyiyeIEEIIBdXqMogVV1zRunTpUuow\nQgihVhkzZswMM2tb2XZFTSDknaHdhtffN7yu8Ed4nfAueNPyg8xsZmoHcD3eenIOcGSqQliuLl26\nMHp0lXWnHkII9YKkcntAyFfsLKbrgafMbG1gQ7wztLPxfmm64v3x5OoM74HX1+6K9ykyoMixhRBC\nqEDREgj52AzbkhoImdnPqel/T7zuL+lvru54T+BOc6/i/Z0scT/mIYQQlk4xnyBWw7uJ+IektyTd\nlrpcaJe6tSD9zXWG15GFO56aRIEOuCT1kzRa0ujp06cXMfwQQqjfiplANMK7NhhgZhvhHZNV1AS9\nUCd0i9TBNbNBZtbDzHq0bVtpGUsIIYQlVMwEYhLetcNraf5BPMH4Mpd1lP5Oy9s+v2fCThTuATOE\nEEI1KFoCYWb/AyZKWist2gnvBG840Dst6413akZa3ktuc2B2LisqhBBC9St2O4gTgXtS99Kf4cNj\nNgCGSuqLjxqV6774CbyK6zi8mmufYgb23XfQsmXl24UQQn1V1ATCzMYCPQqs2qnAtgYcX8x48h1w\nAPz4I1x3HXTrVl2fGkIItUe97GrDDPbaCz78ELbeGp6p1n44QwihdqiXCYQEJ54Ib74Jq6/uicXd\nd5c6qhBCqFnqZQKR06EDvPACbLMNXHgh/PBDqSMKIYSao1Z31lcVllsOnnwSvvwSmjeHuXOhQQN/\nhRBCfRb/BoGmTWHllb1s4thj4ZBD4KeKRvkNIYR6IBKIMtZZB4YOhd13h9mzSx1NCCGUTiQQeSQ4\n4wwvsH7pJS+bmDy51FGFEEJpRAJRwGGHwRNPwPjxsOuuMG9eqSMKIYTqV+8Lqcuzyy5ew2nGDGjY\nsNTRhBBC9YsEogIbbbRgeuBAaN8eevYsXTwhhFCdIospg7lzYcgQ+P3vYdCgUkcTQgjVIxKIDBo1\nguee85pNxx7rjepskZEqQgihbokEIqMWLeCRR6BPH7j4Yk8oIpEIIdRlUQaxGBo3httvh44dvQW2\nCo2BF0IIdUQkEItJgksuWTD/2mve4d+KK5YuphBCKIbIYloKc+Z4raatt4YJE0odTQghVK1IIJbC\nMsvAAw94R39bbAFjx5Y6ohBCqDqRQCylbbbxbjkaNYJtt4WRI0sdUQghVI1IIKrAeuvBK6/AKqvA\nXXeVOpoQQqgaUUhdRTp1ghdf9GwngG++gWWXLW1MIYSwNOIJogq1bg1NmsDMmbDJJt4z7Pz5pY4q\nhBCWTCQQRbDsst7Z3zXXwBFHwM8/lzqiEEJYfJHFVAQNG8KNN3qDunPPhWnT4KGHIssphFC7xBNE\nkUhwzjkweDCMGgWnnFLqiEIIYfHEE0SR9e7tTxIbbODz48Z5q+vWrUsbVwghVCaeIKrBzjtDu3Y+\n3b+/JxjHHANvvlnauEIIoSJFTSAkTZD0rqSxkkanZW0kPSvpk/R3+bRckm6QNE7SO5K6FzO2Urny\nSjj0ULj3Xth4Y9hsMxg2rNRRhRDCoqrjCWIHM+tmZj3S/NnACDPrCoxI8wB7AF3Tqx8woBpiq3bd\nu8Ott8LkyXD99d5e4pNPfN2PP8LHH5c2vhBCyClFFlNPYEiaHgLsl7f8TnOvAq0ltS9BfNWidWs4\n6SR4/33/CzB0KKy1lmdJPfywj2QXQgilUuwEwoBnJI2R1C8ta2dmUwHS35XS8o7AxLz3TkrL6jTJ\nG9cB7LYbXHqpP1Hsv7933XHRRdGOIoRQGsVOILYys+549tHxkratYNtCw+8sMmabpH6SRksaPX36\n9KqKs0Zo1w7+/Gf47DMYPhx++1svn2jc2Nd/+GG0zA4hVJ+iJhBmNiX9nQb8C9gU+DKXdZT+Tkub\nTwI65729EzClwD4HmVkPM+vRtm3bYoZfMg0bwj77wJNPeieAkpdV9OgBa6/tLbS/+qrUUYYQ6rqi\nJRCSWkhqlZsGdgXeA4YDvdNmvYFcHZ7hQK9Um2lzYHYuK6o+a9bM/zZtCgMHQtu23sdTx47exiJX\nwB1CCFWtmE8Q7YCXJL0NvA48bmZPAVcAu0j6BNglzQM8AXwGjANuBf5YxNhqnaZN4fDD4eWX4e23\noU8fL8j+4QdfP3UqfP99aWMMIdQtMlskm7/W6NGjh40ePbrUYZTMnDkLuhc//HB49FHo1csb4627\nbmljCyHUXJLG5DU9KFe0pK7FcokDwPHHe7nFoEE+gNF223lBdwghLKlIIOqILbaAu++GSZPgiitg\n4kQfChW85tMXX5Q2vhBC7VNuFpOkGylQzTTHzE4qVlBZ1fcsporMn+/lEy1awNNPwx57wF57efbT\nbrt5TakQQv1UFVlMo4ExQDOgO/BJenUD5lVFkKF4GjTwxAFg/fV9XIo33vBEYo01/CkjV8AdQgiF\nVFpILWkUsKuZ/ZLmGwPPmNkO1RBfheIJYvH8/DM88ggMGODdjo8fD40aeb9QHTp4e4sQQt1XlYXU\nHYBWefMt07JQyzRpAgcd5AMYvfuuJw6//OLjZ//2t3Dzzd4gL4QQIFsCcQXwlqTBkgYDbwKXFTWq\nUHS5AYvM4OKLPfE4/nhvgHfccdEAL4SQIYEws38Am+FdZfwL2MLMhlT8rlBbNGkCRx8No0fDa6/B\nAQfAkCHw+ee+ftYs74Y8hFD/ZK3m2hCYDswE1qyk071QC0mw6abwj394mcSOO/ryiy+GTp3grLPg\n009LG2MIoXpVOia1pCuBg4H/Arm+RA14oYhxhRJq02bBdM+e/jTxt7/BVVd5FdmTToI99yxdfCGE\n6lFpAoEP6LOWmf1U7GBCzbPddv6aPBluu81bag8duiCBmDEDVlyxtDGGEIojSxbTZ0DjYgcSaraO\nHeHCCxc8TQC89Ra0b7+gZlQt7tYrhFBAlgRiDjBW0i2Sbsi9ih1YqJkaNVqQBdW2rWc3Pfecl1ms\nu66Psx29yoZQN2RpKNe70PKaUJMpGsrVDD/84NlOAwbABx94dlTLljBzJiy/fKmjCyGUlbWhXKbu\nviU1AdZMsx/lWlWXWiQQNc+UKd4q28y7+GjRwvt/OvjghXufDSGUTpW1pJa0Pd4H003AzcDHUc01\nlKdDamM/b543uPvuOzjqKC/DOPXUqCobQm2SpQziGrwvpu3MbFtgN+Da4oYVartGjeDEE+G//4Xn\nn/fqsTfd5I3xwAc7+qVGPIeGEMqTJYFobGYf5WbM7GOiVlPISPJqsvff72NU7L+/L7/hBlhlFa8Z\nNWlSaWMMIRSWJYEYLel2Sdun1614N+AhLJZ27XxsbYDNNoONNoJLLoEuXeB3v4Nnny1peCGEMrI0\nlOsPHA+cBAhvQX1zMYMKdd8OO/hr/Hi45Ra4/Xb46SfYZRdf//33C8azCCGURpZqri2AH81sXppv\nCDQ1sznVEF+FohZT3fHTT94qu2NHHx51nXU8O6p/f9h88xirIoSqVJXjQYwAmufNNweeW9LAQiik\naVNPHMBHwzvySB/caMstPSvqlluiAV4I1S1LAtHMzL7LzaTpqNEeiqZTJ6/xNHkyDBzobSpOOGHB\nYEYxVGoI1SNLAvG9pO65GUkbA/EVDUXXqhUceyyMHevVZdu39+V77bWgZtTPP5c2xhDqsiwJxCnA\nA5JelPQi8E/ghOKGFcICEqyZ2vGbeU+ykybBIYdA585w7rkLBjgKIVSdLCPKvQGsjddm+iOwjplF\nNddQEhKccYYPifrUU16AfeWV8PDDvv6XX7wVdwhh6WXpamMZ4E/AyWb2LtBF0t5FjyyECjRo4K2z\nhw3zqrJ9+/ryu++GNdaAK66AadNKG2MItV2WLKZ/AD8DW6T5ScClWT9AUkNJb0l6LM2vKuk1SZ9I\n+mfqCBBJTdP8uLS+y2IdSai3Vl4Zll12wfSqq8I553hh96GHwosvxlgVISyJLAnE6mb2V+AXADP7\nAW8wl9XJwAd581cC15pZV3yM6/Tbj77ATDNbA+/r6crF+IwQANhpJxg5Et5/39tQPPEEnHLKgvXR\n/1MI2WVJIH6W1BwfhxpJqwOZhh+V1AnYC7gtzQvYEXgwbTIEH9IUoGeaJ63fKW0fwmJbZx0fvGjy\nZLjvPi+7mDXLnypyNaNCCBXLkkBcCDwFdJZ0D95w7qyM+78ubTs/za8AzDKzuWl+EpCaR9ERmAiQ\n1s9O2y9EUj9JoyWNnj59esYwQn3VosWCGlBz5sDee8Odd3rjuy228OkffyxtjCHUVFlqMT0L/B44\nErgP6GFmz1f2vlSQPa1MjadCTwSWYV1+PIPMrIeZ9Wjbtm1lYYTwqw4dvM+nKVPg2mvh66+hd+8F\nY1TMnVvx+0Oob7LUYtoK74vpcaA1cK6kVTLseytgX0kTgPvxrKXrgNaScp0EdgKmpOlJQOf0mY2A\n5YCvsx9KCNksv7yXS3z4Ibz+Oqy3ni8/4givGfXII5FYhADZspgGAHMkbQicCXwO3FnZm8zsHDPr\nZGZdgD8AI83sMGAUcEDarDcwLE0PT/Ok9SMty3ioISwhCTbZZMH8hht6i+3f/c5rQl18sT9thFBf\nZUkg5qZ/1D2BG8zseqDVUnzmn4DTJI3DyxhuT8tvB1ZIy08Dzl6KzwhhsZ19NkyYAP/6F6y7rg9m\ndOONvm7+/KgqG+qfLN19/xsvpO4DbAtMB8aa2QbFD69i0d13KKZx46BlS/jNb+Dxx+H0032c7d69\nPZsqhNqqKrv7Phiv1trXzP6H1za6ainjC6HGW2MNTxwAmjeHNm3g1FO9W/KjjoI33ihtfCEUW6VP\nEDVZPEGE6jZ2LAwYAPfcAyusAJ99Bg0behZUgyw/t0KoAaryCSKEkHTr5oMXTZ7sHQQ2bOhdjq+7\nLpx8steMCqGuiAQihCWw3HKw8cY+PWsWdO/uTxbrrAM77ggPPBDdeoTar9wEQtIgSb+TtDQ1lkKo\n81ZaCe6918eouPxy7132oIPgP//x9bU4FzfUcxU9QdwBbAg8IWmEpD+lthAhhAJWWsmryo4bB88+\nC9tu68tPPx169vTxK+bPr3gfIdQk5SYQZvaqmV1kZtsABwFfAKenrrvvkHRQtUUZQi3SsCHsvLM3\nxANo2xZefRX22MP7hbrqKpgxo7QxhpBFpjIIM/vKzO4zs15mthFwE9C1uKGFUDeccw5MnOi9ynbs\nCGed5ctyIgsq1FRLVEhtZmPM7C9VHUwIdVWTJvCHP8C//w3vvutZUeBtKbp1g4ED4dtvSxtjCGVF\nLaYQqtn668Pqq/v09997VlT//v50cfzx8N57pY0vhJxIIEIooe23h7fe8hpP++3n3ZFvs02MURFq\nhizdfR+Yq+oq6TxJD0vqXvzQQqgfpAWDF02eDA89BM2aednEdtt5dtT48aWOMtRHWZ4gzjezbyVt\nDeyGDws6oLhhhVA/rbCCN7QD+OYb7//pqqs8S2rPPeHRR2HevNLGGOqPLAlE7nbcCxhgZsOAJsUL\nKYQA3lr7X//yLsjPO8/7gdp3Xxg2rNK3hlAlsiQQkyXdgreFeEJS04zvCyFUgc6dffCizz/37Ke9\n9/blf/3rgppRUVU2FEOWf/QHAU8Du5vZLKANPrJcCKEaNW4Mv/+9V5kFz2p6+mkv6F5/fR/caPbs\nkoYY6phKEwgzmwNMA7ZOi+YCnxQzqBBC5c45xwu177gDWrSAk07ycSpCqCpZajFdiA8Tmmv72Ri4\nu5hBhRCyWWYZ6NMHXn8dRo+G88/35ePHe82oIUPghx9KG2OovbJkMf0O2Bf4HsDMprB0Y1KHEIpg\n4429VTbAlCkwcyYceaQ3wDv9dPgknvvDYsqSQPxsPuycAUhqUdyQQghLa6ut4IMPYORI7zjwhhtg\ngw2ijCIsnkYZthmaajG1lnQMcBRwa3HDCiEsLQl22MFfU6fCK6941VmAQw6BtdeGY46BDh1KG2eo\nubIUUl8NPAg8BKwFXGBmNxY7sBBC1Wnf3mtAgXfjMXMmXHQRrLwy7L8/PPdcjFURFpWlkHpV4EUz\nO9PMzgBektSl2IGFEIqjWTMfvGjcODjtNG9HscsuMHhwqSMLNU2WMogHgPzfFvPSshBCLbb66t7Y\nbtIkuOsuf5IA7xMqVzMqGuDVb1kSiEZm9nNuJk1HVxsh1BHNmsHhhy8on/jf/+CBB2CzzaBHD7jt\nNu+WPNQ/WRKI6ZL2zc1I6gnEgIkh1FFnneXVZG+6CX7+2Quy99238veFukdWyTOkpNWBe4AOgICJ\nQC8zG1f88CrWo0cPGz16dKnDCKHOMoOXX/ZuPbbbDr7+2mtA9e3r41c0ibyEWknSGDPrUdl2lVZz\nNbNPgc0ltcQTlEwDI0pqBrwANE2f86CZXZgKve/H+3R6EzjCzH5OnQDeCWwMfAUcbGYTsnxWCKE4\nJNh66wXzn34KH38MBx8M7drB0UdDv35eGyrUPVlqMTWVdChwEnCqpAskXZBh3z8BO5rZhkA3YHdJ\nmwNXAteaWVdgJtA3bd8XmGlmawDXpu1CCDXIJpt47afHH/fpyy6D1VbzLKlQ92QpgxgG9MQ76fs+\n71Uhc9+l2cbpZcCOeLsK8MGH9kvTPdM8af1OkpQhvhBCNWrYcMHgRePHw4ABCxrbnXYaXHklTJ9e\n2hhD1cjSkrqTme2+JDuX1BAYA6wB3AR8Cswys7lpk0lAxzTdES/fwMzmSpoNrECZAnFJ/YB+ACvH\nc20IJbXKKl6IDV5O8e67cO21cMEFcMAB0L+/d/sRP/VqpyxPEP+RtMGS7NzM5plZN6ATsCmwTqHN\n0t9Ct9AiJehmNsjMephZj7ZkMdB2AAAV1UlEQVRt2y5JWCGEImjYEJ59Ft57z8slHnsMttkGrr++\n1JGFJZUlgdgaGCPpI0nvSHpX0juL8yFpoKHngc3xPp1yTy6dgFzu5SSgM0Bavxzw9eJ8Tgih9NZb\nzwcvmjwZBg1a0ADvySf9ieKdxfrvEUopSwKxB9AV2BXYB9g7/a2QpLaSWqfp5sDOwAfAKOCAtFlv\nvIwDYHiaJ60faZXVwQ0h1FgtW3r2U+fOPv/hh96dx4YberbT3Xd7v1Ch5srSWd/nZvY58AOe5fNr\n19+VaA+MSk8bbwDPmtlj+OBDp0kah5cx3J62vx1YIS0/DTh7cQ8mhFBznXqqd+tx9dUwbRoccYS3\nrQg1V5aGcvsC1+AN5aYBqwAfmNl6xQ+vYtFQLoTaaf58GDHCu/DYbz9/kujd27v82HNPL88IxZO1\noVyWLKZL8LKDj81sVWAn4OWljC+EUI81aOA9yO6XKrl/8gm89JJ36bHaavCXv3ifUKG0siQQv5jZ\nV0ADSQ3MbBTe8C2EEKrEBhvAhAnw0EPQtSucd56XXXz0Uakjq9+ytIOYlbrZeAG4R9I0vNFcCCFU\nmcaNfVCj3//eu/N46CFYc01fd+WV0Lw59OoFrVuXNs76JMsTRE9gDnAq8BTe2G3vYgYVQqjf1lwT\nzjnHG9iZ+Yh3J58MHTt6zag33yx1hPVDlgTiAjObb2ZzzWyImd2A10QKIYSik7wB3pgxcOihcO+9\nsPHG3g9UKK4sCcQuBZbtUdWBhBBCRbp3h1tv9QZ411+/YIyKN97wPqA+/ri08dVF5SYQkvpLehdY\nO7Wgzr3GA9EWMoRQEq1bw0knwfrr+/wbb3jL7bXW8ppRDz8Mc6OUtEqU2w5C0nLA8sDlLNxo7Vsz\nqxFdYEQ7iBACeJXY22/3rj2++MJrRb39dnQSWJ6lbgdhZrPTgD3nAf9LralXBQ7PdaERQgg1wW9+\nA3/+M3z2GQwfDqec4onD/Pnwxz96Gcb8+aWOsvbJUgbxEDBP0hp4dxirAvcWNaoQQlgCDRvCPvvA\nUUf5/Pjx8MADsOuungV1zTXw1VeljbE2yZJAzE/jN/weuM7MTsX7WQohhBpt9dW9/6e77/YhUs84\nw6vKRs50NplaUks6BOgFPJaWNS5eSCGEUHWaNoXDDvOuPN5+G0480XuUBS+3uPVW7xMqLCpLAtEH\n2AL4i5mNl7QqcHdxwwohhKr329/CVVd5q23w1tr9+vmQqSeeCO+/X9r4apos3X2/b2Ynmdl9aX68\nmV1R/NBCCKG4Hn8cXn7Z21QMGuSDHZ1/fqmjqjkqTSAkbSXpWUkfS/pM0nhJn1VHcCGEUEwSbLkl\n3HWXl1VceSXstpuvGzfOa0Z9/nlpYyylLONBfIj3wzQGmJdbnnp4LaloBxFCKJY77vB+n8DHqOjf\n3xOPujBWRVWOBzHbzJ40s2lm9lXuVQUxhhBCjXXUUV5N9txzvbX2Xnt5A7z61Eo7SwIxStJVkraQ\n1D33KnpkIYRQYiuvDJdc4q2z//lPH/WuURok4cILvWZUJZkwtVqWLKZRBRabme1YnJCyiyymEEIp\nfPmlN7ybPdv7hOrf34dLXXbZUkeWTZVlMZnZDgVeJU8cQgihVNq1815lb7sNmjSB44/3BngvvFDq\nyKpWuSPKSTrczO6WdFqh9Wb2t+KFFUIINVuLFtC3r5dVvPGGJxYbbeTrhg2Db76BAw+EZs1KG+fS\nqOgJokX626qcVwgh1HsSbLqpt6Nolf4zDh7sw6N26gRnnQWfflrSEJdYpWUQNVmUQYQQaqL582Hk\nSBgwwJ8m5s2DM8+Ev/611JG5qqzmGkIIYTE0aAA77+xdeXz+OVx0EWy1la+bPt1rRk2dWtIQM4kE\nIoQQiqhjR68S27Onz48YARdc4FVoDzwQRo2quVVlI4EIIYRq9Ic/+PjZJ5/s2VA77uh9QM2ZU+rI\nFpUpgSjbMC5LQzlJnSWNkvSBpP9KOjktb5P6dvok/V0+LZekGySNS2NfR2O8EEKd1LUrXH219/80\nZAjssQcss4yv+/vfa854FVmfIPpXMl/IXOB0M1sH2Bw4XtK6+PjWI8ysKzCCBeNd7wF0Ta9+wICM\nsYUQQq3UvLnXdrrmGp//9lvvIHCTTfx1xx2lfbLIlECY2TEVzZfznqlm9maa/hb4AOgI9ASGpM2G\nAPul6Z7AneZeBVpLipHrQgj1RqtW3q3HjTd6wtC3r5dhPP10aeKpqKFchVk8uX/+WUjqAmwEvAa0\nM7OpaR9TJa2UNusITMx726S0rBaU9YcQQtVYbjk44QRvnf3iizBwoHcSCN5Se9o0L/BuXA3jepab\nQADpoYdmQA/gbUDAb/F/9Ftn+QBJLYGHgFPM7BtJ5W5aYNkiZfuS+uFZUKy88spZQgghhFpHgm23\n9VfOLbfAvffC9tt77adiKzeBMLMdACTdD/Qzs3fT/PrAGVl2LqkxnjjcY2YPp8VfSmqfnh7aA9PS\n8klA57y3dwKmFIhrEDAIvKFcljhCCKEuuPNOOOQQb4hXHbKUQaydSxwAzOw9oFtlb5I/KtwOfFCm\n36bhQO803RsYlre8V6rNtDk+DkVkL4UQQtKwIey9tw+RWh0qymLK+UDSbcDdeJbP4XiBc2W2Ao4A\n3pU0Ni07F7gCGCqpL/AFcGBa9wSwJzAOmAP0yXoQIYQQql6WBKIPXq315DT/AhmqoJrZSxQuVwDY\nqcD2BhyfIZ4QQgjVoNIEwsx+lDQQeMLMPqqGmEIIIdQAlZZBSNoXGAs8lea7SRpe7MBCCCGUVpZC\n6guBTYFZAGY2FuhSxJhCCCHUAFkSiLlmNrvokYQQQqhRshRSvyfpUKChpK7AScB/ihtWCCGEUsvy\nBHEisB7wE3AvMBs4pZhBhRBCKL0stZjmAH+WdJmZfV8NMYUQQqgBstRi2lLS+6TGcZI2lHRz0SML\nIYRQUlmymK4FdgO+AjCzt4FtK3xHCCGEWi/reBATyyyaV4RYQggh1CBZajFNlLQlYJKa4LWYsvTF\nFEIIoRbL8gRxHN5HUkdgMt6Ta/SZFEIIdVyWWkwzgMOqIZYQQgg1SJZaTKtJelTSdEnTJA2TtFp1\nBBdCCKF0smQx3QsMBdoDHYAHgPuKGVQIIYTSy5JAyMzuMrO56ZUbOCiEEEIdlqUW0yhJZwP34wnD\nwcDjktoAmNnXRYwvhBBCiWRJIA5Of48ts/woPMGI8ogQQqiDstRiWrU6AgkhhFCzZKnFdKCkVmn6\nPEkPS9qo+KGFEEIopSyF1Oeb2beStsb7ZBoCDCxuWCGEEEotSwKR63dpL2CAmQ0DmhQvpBBCCDVB\nlgRisqRbgIOAJyQ1zfi+EEIItViWf/QHAU8Du5vZLKANcGZRowohhFByWUeUezhvfiowtZhBhRBC\nKL3IKgohhFBQJBAhhBAKKloCIemO1Pvre3nL2kh6VtIn6e/yabkk3SBpnKR3JHUvVlwhhBCyKeYT\nxGBg9zLLzgZGmFlXYESaB9gD6Jpe/YABRYwrhBBCBkVLIMzsBaBsR3498YZ2pL/75S2/09yrQGtJ\n7YsVWwghhMpVdxlEu1QLKlcbaqW0vCMwMW+7SWnZIiT1kzRa0ujp06cXNdgQQqjPakohtQosKzjm\nhJkNMrMeZtajbdu2RQ4rhBDqr+pOIL7MZR2lv9PS8klA57ztOgFTqjm2EEIIeao7gRgO9E7TvYFh\nect7pdpMmwOzc1lRIYQQSiPLgEFLRNJ9wPbAipImARcCVwBDJfUFvgAOTJs/AewJjAPmAH2KFVcI\nIYRsipZAmNkh5azaqcC2BhxfrFhCCCEsvppSSB1CCKGGiQQihBBCQZFAhBBCKKhoZRA13vbbL7rs\noIPgj3+EOXNgzz0XXX/kkf6aMQMOOGDR9f37w8EHw8SJcMQRi64//XTYZx/46CM49thF1593Huy8\nM4wdC6ecsuj6yy6DLbeE//wHzj130fXXXQfdusFzz8Glly66/pZbYK214NFH4ZprFl1/113QuTP8\n858woEBvJw8+CCuuCIMH+6usJ56AZZaBm2+GoUMXXf/88/736qvhsccWXte8OTz5pE9fcgmMGLHw\n+hVWgIce8ulzzoFXXll4fadOcPfdPn3KKX4O8625Jgwa5NP9+sHHHy+8vls3P38Ahx8OkyYtvH6L\nLeDyy316//3hq68WXr/TTnD++T69xx7www8Lr997bzjjDJ+Oe2/R9XHv+fTi3Hu5YyqieIIIIYRQ\nkLwCUe3Uo0cPGz16dKnDCCGEWkXSGDPrUdl28QQRQgihoEggQgghFBQJRAghhIIigQghhFBQJBAh\nhBAKigQihBBCQZFAhBBCKCgSiBBCCAVFAhFCCKGgSCBCCCEUFAlECCGEgiKBCCGEUFAkECGEEAqK\nBCKEEEJBkUCEEEIoKBKIEEIIBUUCEUIIoaBIIEIIIRQUCUQIIYSCIoEIIYRQUCQQIYQQCqpRCYSk\n3SV9JGmcpLNLHU8IIdRnNSaBkNQQuAnYA1gXOETSuqWNKoQQ6q8ak0AAmwLjzOwzM/sZuB/oWeKY\nQgih3mpU6gDydAQm5s1PAjYru5GkfkC/NPudpI+W4LNWBGYswftqszjm+iGOuf5YmuNeJctGNSmB\nUIFltsgCs0HAoKX6IGm0mfVYmn3UNnHM9UMcc/1RHcddk7KYJgGd8+Y7AVNKFEsIIdR7NSmBeAPo\nKmlVSU2APwDDSxxTCCHUWzUmi8nM5ko6AXgaaAjcYWb/LdLHLVUWVS0Vx1w/xDHXH0U/bpktks0f\nQggh1KgsphBCCDVIJBAhhBAKqlcJRF3tykNSZ0mjJH0g6b+STk7L20h6VtIn6e/yabkk3ZDOwzuS\nupf2CJacpIaS3pL0WJpfVdJr6Zj/mSo8IKlpmh+X1ncpZdxLQ1JrSQ9K+jBd8y3q+rWWdGq6t9+T\ndJ+kZnXtWku6Q9I0Se/lLVvs6yqpd9r+E0m9lyamepNA1PGuPOYCp5vZOsDmwPHp2M4GRphZV2BE\nmgc/B13Tqx8woPpDrjInAx/kzV8JXJuOeSbQNy3vC8w0szWAa9N2tdX1wFNmtjawIX78dfZaS+oI\nnAT0MLP18Uosf6DuXevBwO5lli3WdZXUBrgQb2S8KXBhLlFZImZWL17AFsDTefPnAOeUOq4iHesw\nYBfgI6B9WtYe+ChN3wIckrf9r9vVphfeVmYEsCPwGN7YcgbQqOw1x2vHbZGmG6XtVOpjWIJjXhYY\nXzb2unytWdDLQpt07R4DdquL1xroAry3pNcVOAS4JW/5Qtst7qvePEFQuCuPjiWKpWjS4/RGwGtA\nOzObCpD+rpQ2qyvn4jrgLGB+ml8BmGVmc9N8/nH9esxp/ey0fW2zGjAd+EfKWrtNUgvq8LU2s8nA\n1cAXwFT82o2h7l9rWPzrWqXXuz4lEJm68qjNJLUEHgJOMbNvKtq0wLJadS4k7Q1MM7Mx+YsLbGoZ\n1tUmjYDuwAAz2wj4ngXZDoXU+uNOWSQ9gVWBDkALPIulrLp2rStS3jFW6bHXpwSiTnflIakxnjjc\nY2YPp8VfSmqf1rcHpqXldeFcbAXsK2kC3vPvjvgTRWtJuQag+cf16zGn9csBX1dnwFVkEjDJzF5L\n8w/iCUZdvtY7A+PNbLqZ/QI8DGxJ3b/WsPjXtUqvd31KIOpsVx6SBNwOfGBmf8tbNRzI1WLojZdN\n5Jb3SjUhNgdm5x5jawszO8fMOplZF/xajjSzw4BRwAFps7LHnDsXB6Tta92vSjP7HzBR0lpp0U7A\n+9Tha41nLW0uaZl0r+eOuU5f62Rxr+vTwK6Slk9PXrumZUum1IUy1VwAtCfwMfAp8OdSx1OFx7U1\n/hj5DjA2vfbE811HAJ+kv23S9sJrdH0KvIvXDin5cSzF8W8PPJamVwNeB8YBDwBN0/JmaX5cWr9a\nqeNeiuPtBoxO1/sRYPm6fq2B/wM+BN4D7gKa1rVrDdyHl7H8gj8J9F2S6woclY59HNBnaWKKrjZC\nCCEUVJ+ymEIIISyGSCBCCCEUFAlECCGEgiKBCCGEUFAkECGEEAqKBCLUWpK2ST18jpW0jqRDSx1T\nsUjaPtdj7VLu5wlJrasiplD3RQIRarPDgKvNrBvQDqizCURVMbM9zWxWqeMItUMkEKHGkNRC0uOS\n3k79/h+clu+UOqZ7N/WZ31TS0cBBwAWS7gGuALZJTxOnSjpS0iOSHpU0XtIJkk5L+3k1dYuMpGMk\nvZE+8yFJy6TlwyT1StPHps8oG++BKc63Jb2QlnWR9KKkN9Nry7R8e0n/ljRU0seSrpB0mKTX03Gt\nnrYbLGlg2sfHqc+pQufpjhT3W5J6FtimvaQX0vl4T9I2afkESStKOi6tG5vOz6i0fldJr6TYH0j9\ne4X6qtStB+MVr9wL2B+4NW9+ObxV7ERgzbTsTrwzQvD+8w9I09uTWlOn+SPxlqStgLZ4j57HpXXX\n5u1jhbz3XAqcmKbbpfdvg7e+b1Mg3neBjmm6dfq7DNAsTXcFRufFNwvvkrkpMBn4v7TuZOC6vGN6\nCv/x1hVvUduMhVuLXwYcnvvcFF+LMrGdTuotAB8/oVWangCsmLddY+BFYB9gReCF3L6APwEXlPq+\niFfpXvEEEWqSd4GdJV0paRszmw2shXfU9nHaZgiwbcb9jTKzb81sOp5APJr3OV3S9Prp1/q7eJbV\negBm9iVwAd7fz+lmVqizt5eBwZKOwf8Jg//DvTXt7wF8cKqcN8xsqpn9hHeR8EyBeACGmtl8M/sE\n+AxYu8zn7gqcLWks8DyegKxcZps3gD6SLgI2MLNvC8QPPvjQSDN7FB9sal3g5bTv3sAq5bwv1AON\nKt8khOphZh9L2hjvR+pySc+wdB0q/pQ3PT9vfj4L7v3BwH5m9rakI/Ff6jkbAF/hXUwXivc4SZsB\newFjJXUDTgS+xEd6awD8uJjxwKLdM5edF7C/mX1UKK4U2wuStk2x3SXpKjO7c6Gd+PGuApyQt99n\nzeyQ8vYb6pd4ggg1hqQOwBwzuxsfIKY73kFbF0lrpM2OAP5d4O3f4tlJi6sVMFXeXfphebFsio85\nsBFwhqRVC8S7upm9ZmYX4KOWdcazxaaa2fwUa8Oy78vgQEkNUrnEavhoYfmeBk5MPZsiaaMCsa2C\nj5dxK97Tb/cy6zcGzsCzqnIDLr0KbJU71/LeU9dcgvhDHRFPEKEm2QC4StJ8vEfL/mb2o6Q+wAPy\nvv3fAAYWeO87wFxJb+NPBTMzfub5+Oh7n+NZPa0kNQVuxXvCnCLpdOAOSTuaWf6v+askdcV/eY8A\n3gZuBh6SdCCePfX9Yhx/zkd4ItgOLzf5MaUFOZfgY1+8kxKJCUDZwuztgTMl/QJ8B/Qqs/4EfAjP\nUWnfo83s6PRUcV86BwDn4WUcoR6K3lxDqEEkDcYLox8sdSwhRBZTCCGEguIJIoQQQkHxBBFCCKGg\nSCBCCCEUFAlECCGEgiKBCCGEUFAkECGEEAr6f+LLgLu2fiDxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266ac316ef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"start...\")\n",
    "\n",
    "# Reset parameters\n",
    "class Param:\n",
    "    # Learning parameters\n",
    "    learning_rate = 0.03\n",
    "    minibatch_size = 8\n",
    "    num_minbatches = 100\n",
    "    test_set_size = 1 # we are only interrested in speed\n",
    "    momentum = 0.8187307530779818\n",
    "    reporting_interval = 1000000 # Switch off reporting to speed up\n",
    "    allow_duplicates = False\n",
    "    \n",
    "    # Parameters for sampled softmax\n",
    "    use_sampled_softmax = True\n",
    "    use_sparse = True\n",
    "    softmax_sample_size = 10\n",
    "\n",
    "    # Details of data and model\n",
    "    num_classes = 50000\n",
    "    hidden_dim = 10\n",
    "    \n",
    "data_sampling_distribution = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)\n",
    "softmax_sampling_weights = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)\n",
    "\n",
    "    \n",
    "sample_sizes = [5, 10, 100, 1000]\n",
    "speed_with_sampled_softmax = []\n",
    "\n",
    "# Get the speed with sampled softmax for different sizes\n",
    "for sample_size in sample_sizes: \n",
    "    print(\"Measuring speed of sampled softmax for sample size %d ...\" % (sample_size))\n",
    "    Param.use_sampled_softmax = True\n",
    "    Param.softmax_sample_size = sample_size\n",
    "    _, _,  samples_per_second = train(do_print_progress = False)\n",
    "    speed_with_sampled_softmax.append(samples_per_second)\n",
    "\n",
    "# Get the speed with full softmax\n",
    "Param.use_sampled_softmax = False\n",
    "print(\"Measuring speed of full softmax ...\")\n",
    "_, _,  samples_per_second = train(do_print_progress = False)\n",
    "speed_without_sampled_softmax = np.repeat(samples_per_second, len(sample_sizes))\n",
    "\n",
    "# Plot the speed of sampled softmax (blue) as a function of sample sizes\n",
    "# and compare it to the speed with full softmax (red).    \n",
    "plt.plot(sample_sizes, speed_without_sampled_softmax, 'r--',sample_sizes, speed_with_sampled_softmax, 'b--')\n",
    "plt.xlabel('softmax sample size')\n",
    "plt.ylabel('speed: instances / second')\n",
    "plt.title(\"Speed 'sampled softmax' (blue) vs. 'full softmax' (red)\")\n",
    "plt.ylim(ymin=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}