800 строки
66 KiB
Plaintext
800 строки
66 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from IPython.display import Image"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"nbpresent": {
|
|
"id": "29b9bd1d-766f-4422-ad96-de0accc1ce58"
|
|
}
|
|
},
|
|
"source": [
|
|
"# CNTK 103: Part C - Multi Layer Perceptron with MNIST\n",
|
|
"\n",
|
|
"We assume that you have successfully completed CNTK 103 Part A.\n",
|
|
"\n",
|
|
"In this tutorial, we train a multi-layer perceptron on MNIST data. This notebook provides the recipe using Python APIs. If you are looking for this example in BrainScript, please look [here](https://github.com/Microsoft/CNTK/tree/v2.0/Examples/Image/GettingStarted)\n",
|
|
"\n",
|
|
"## Introduction\n",
|
|
"\n",
|
|
"**Problem** \n",
|
|
"As in CNTK 103B, we will continue to work on the same problem of recognizing digits in MNIST data. The MNIST data comprises hand-written digits with little background noise."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<img src=\"http://3.bp.blogspot.com/_UpN7DfJA0j4/TJtUBWPk0SI/AAAAAAAAABY/oWPMtmqJn3k/s1600/mnist_originals.png\" width=\"200\" height=\"200\"/>"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Image object>"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Figure 1\n",
|
|
"Image(url= \"http://3.bp.blogspot.com/_UpN7DfJA0j4/TJtUBWPk0SI/AAAAAAAAABY/oWPMtmqJn3k/s1600/mnist_originals.png\", width=200, height=200)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Goal**:\n",
|
|
"Our goal is to train a classifier that will identify the digits in the MNIST dataset. Additionally, we aspire to achieve lower error rate with Multi-layer perceptron compared to Multi-class logistic regression. \n",
|
|
"\n",
|
|
"**Approach**:\n",
|
|
"The same 5 stages we have used in the previous tutorial are applicable: Data reading, Data preprocessing, Creating a model, Learning the model parameters and Evaluating (a.k.a. testing/prediction) the model. \n",
|
|
"- Data reading: We will use the CNTK Text reader \n",
|
|
"- Data preprocessing: Covered in part A (suggested extension section). \n",
|
|
"\n",
|
|
"There is a high overlap with CNTK 102. Though this tutorial we adapt the same model to work on MNIST data with 10 classes instead of the 2 classes we used in CNTK 102.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {
|
|
"collapsed": true,
|
|
"nbpresent": {
|
|
"id": "138d1a78-02e2-4bd6-a20e-07b83f303563"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)\n",
|
|
"import matplotlib.image as mpimg\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"import sys\n",
|
|
"import os\n",
|
|
"\n",
|
|
"import cntk as C\n",
|
|
"import cntk.tests.test_utils\n",
|
|
"cntk.tests.test_utils.set_device_from_pytest_env() # (only needed for our build system)\n",
|
|
"C.cntk_py.set_fixed_random_seed(1) # fix a random seed for CNTK components\n",
|
|
"\n",
|
|
"%matplotlib inline"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Data reading\n",
|
|
"\n",
|
|
"In this section, we will read the data generated in CNTK 103 Part A."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define the data dimensions\n",
|
|
"input_dim = 784\n",
|
|
"num_output_classes = 10"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"In this tutorial we are using the MNIST data you have downloaded using CNTK_103A_MNIST_DataLoader notebook. The dataset has 60,000 training images and 10,000 test images with each image being 28 x 28 pixels. Thus the number of features is equal to 784 (= 28 x 28 pixels), 1 per pixel. The variable `num_output_classes` is set to 10 corresponding to the number of digits (0-9) in the dataset.\n",
|
|
"\n",
|
|
"The data is in the following format:\n",
|
|
"\n",
|
|
" |labels 0 0 0 0 0 0 0 1 0 0 |features 0 0 0 0 ... \n",
|
|
" (784 integers each representing a pixel)\n",
|
|
" \n",
|
|
"In this tutorial we are going to use the image pixels corresponding the integer stream named \"features\". We define a `create_reader` function to read the training and test data using the [CTF deserializer](https://cntk.ai/pythondocs/cntk.io.html?highlight=ctfdeserializer#cntk.io.CTFDeserializer). The labels are [1-hot encoded](https://en.wikipedia.org/wiki/One-hot). Refer to CNTK 103A tutorial for data format visualizations. \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Read a CTF formatted text (as mentioned above) using the CTF deserializer from a file\n",
|
|
"def create_reader(path, is_training, input_dim, num_label_classes):\n",
|
|
" return C.io.MinibatchSource(C.io.CTFDeserializer(path, C.io.StreamDefs(\n",
|
|
" labels = C.io.StreamDef(field='labels', shape=num_label_classes, is_sparse=False),\n",
|
|
" features = C.io.StreamDef(field='features', shape=input_dim, is_sparse=False)\n",
|
|
" )), randomize = is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Data directory is ..\\Examples\\Image\\DataSets\\MNIST\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Ensure the training and test data is generated and available for this tutorial.\n",
|
|
"# We search in two locations in the toolkit for the cached MNIST data set.\n",
|
|
"data_found = False\n",
|
|
"for data_dir in [os.path.join(\"..\", \"Examples\", \"Image\", \"DataSets\", \"MNIST\"),\n",
|
|
" os.path.join(\"data\", \"MNIST\")]:\n",
|
|
" train_file = os.path.join(data_dir, \"Train-28x28_cntk_text.txt\")\n",
|
|
" test_file = os.path.join(data_dir, \"Test-28x28_cntk_text.txt\")\n",
|
|
" if os.path.isfile(train_file) and os.path.isfile(test_file):\n",
|
|
" data_found = True\n",
|
|
" break\n",
|
|
"if not data_found:\n",
|
|
" raise ValueError(\"Please generate the data by completing CNTK 103 Part A\")\n",
|
|
"print(\"Data directory is {0}\".format(data_dir))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='#Model Creation'></a>\n",
|
|
"## Model Creation\n",
|
|
"\n",
|
|
"Our multi-layer perceptron will be relatively simple with 2 hidden layers (`num_hidden_layers`). The number of nodes in the hidden layer being a parameter specified by `hidden_layers_dim`. The figure below illustrates the entire model we will use in this tutorial in the context of MNIST data.\n",
|
|
"\n",
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you are not familiar with the terms *hidden layer* and *number of hidden layers*, please refer back to CNTK 102 tutorial.\n",
|
|
"\n",
|
|
"Each Dense layer (as illustrated below) shows the input dimensions, output dimensions and activation function that layer uses. Specifically, the layer below shows: input dimension = 784 (1 dimension for each input pixel), output dimension = 400 (number of hidden nodes, a parameter specified by the user) and activation function being [relu](https://cntk.ai/pythondocs/cntk.ops.html?highlight=relu#cntk.ops.relu).\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"In this model we have 2 dense layer called the hidden layers each with an activation function of `relu` and one output layer with no activation. \n",
|
|
"\n",
|
|
"The output dimension (a.k.a. number of hidden nodes) in the 2 hidden layer is set to 400 and 200 in the illustration above. In the code below we keep both layers to have the same number of hidden nodes (set to 400). The number of hidden layers is 2. Fill in the following values:\n",
|
|
"- num_hidden_layers\n",
|
|
"- hidden_layers_dim\n",
|
|
"\n",
|
|
"The final output layer emits a vector of 10 values. Since we will be using softmax to normalize the output of the model we do not use an activation function in this layer. The softmax operation comes bundled with the [loss function](https://cntk.ai/pythondocs/cntk.losses.html) we will be using later in this tutorial."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"num_hidden_layers = 2\n",
|
|
"hidden_layers_dim = 400"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Network input and output: \n",
|
|
"- **input** variable (a key CNTK concept): \n",
|
|
">An **input** variable is a container in which we fill different observations in this case image pixels during model learning (a.k.a.training) and model evaluation (a.k.a. testing). Thus, the shape of the `input` must match the shape of the data that will be provided. For example, when data are images each of height 10 pixels and width 5 pixels, the input feature dimension will be 50 (representing the total number of image pixels). More on data and their dimensions to appear in separate tutorials.\n",
|
|
"\n",
|
|
"\n",
|
|
"**Question** What is the input dimension of your chosen model? This is fundamental to our understanding of variables in a network or model representation in CNTK.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"input = C.input_variable(input_dim)\n",
|
|
"label = C.input_variable(num_output_classes)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Multi-layer Perceptron setup\n",
|
|
"\n",
|
|
"The cell below is a direct translation of the illustration of the model shown above."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def create_model(features):\n",
|
|
" with C.layers.default_options(init = C.layers.glorot_uniform(), activation = C.ops.relu):\n",
|
|
" h = features\n",
|
|
" for _ in range(num_hidden_layers):\n",
|
|
" h = C.layers.Dense(hidden_layers_dim)(h)\n",
|
|
" r = C.layers.Dense(num_output_classes, activation = None)(h)\n",
|
|
" return r\n",
|
|
" \n",
|
|
"z = create_model(input)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"`z` will be used to represent the output of a network.\n",
|
|
"\n",
|
|
"We introduced sigmoid function in CNTK 102, in this tutorial you should try different activation functions in the hidden layer. You may choose to do this right away and take a peek into the performance later in the tutorial or run the preset tutorial and then choose to perform the suggested activity.\n",
|
|
"\n",
|
|
"\n",
|
|
"** Suggested Activity **\n",
|
|
"- Record the training error you get with `sigmoid` as the activation function\n",
|
|
"- Now change to `relu` as the activation function and see if you can improve your training error\n",
|
|
"\n",
|
|
"*Quiz*: Name some of the different supported activation functions. Which activation function gives the least training error?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Scale the input to 0-1 range by dividing each pixel by 255.\n",
|
|
"z = create_model(input/255.0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Learning model parameters\n",
|
|
"\n",
|
|
"Same as the previous tutorial, we use the `softmax` function to map the accumulated evidences or activations to a probability distribution over the classes (Details of the [softmax function][]).\n",
|
|
"\n",
|
|
"[softmax function]: http://cntk.ai/pythondocs/cntk.ops.html#cntk.ops.softmax"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Training\n",
|
|
"\n",
|
|
"Similar to CNTK 102, we minimize the cross-entropy between the label and predicted probability by the network. If this terminology sounds strange to you, please refer to the tutorial CNTK 102 for a refresher. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"loss = C.cross_entropy_with_softmax(z, label)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Evaluation\n",
|
|
"\n",
|
|
"In order to evaluate the classification, one can compare the output of the network which for each observation emits a vector of evidences (can be converted into probabilities using `softmax` functions) with dimension equal to number of classes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"label_error = C.classification_error(z, label)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Configure training\n",
|
|
"\n",
|
|
"The trainer strives to reduce the `loss` function by different optimization approaches, [Stochastic Gradient Descent][] (`sgd`) being a basic one. Typically, one would start with random initialization of the model parameters. The `sgd` optimizer would calculate the `loss` or error between the predicted label against the corresponding ground-truth label and using [gradient-decent][] generate a new set model parameters in a single iteration. \n",
|
|
"\n",
|
|
"The aforementioned model parameter update using a single observation at a time is attractive since it does not require the entire data set (all observation) to be loaded in memory and also requires gradient computation over fewer datapoints, thus allowing for training on large data sets. However, the updates generated using a single observation sample at a time can vary wildly between iterations. An intermediate ground is to load a small set of observations and use an average of the `loss` or error from that set to update the model parameters. This subset is called a *minibatch*.\n",
|
|
"\n",
|
|
"With minibatches we often sample observation from the larger training dataset. We repeat the process of model parameters update using different combination of training samples and over a period of time minimize the `loss` (and the error). When the incremental error rates are no longer changing significantly or after a preset number of maximum minibatches to train, we claim that our model is trained.\n",
|
|
"\n",
|
|
"One of the key parameter for optimization is called the `learning_rate`. For now, we can think of it as a scaling factor that modulates how much we change the parameters in any iteration. We will be covering more details in later tutorial. \n",
|
|
"With this information, we are ready to create our trainer. \n",
|
|
"\n",
|
|
"[optimization]: https://en.wikipedia.org/wiki/Category:Convex_optimization\n",
|
|
"[Stochastic Gradient Descent]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent\n",
|
|
"[gradient-decent]: http://www.statisticsviews.com/details/feature/5722691/Getting-to-the-Bottom-of-Regression-with-Gradient-Descent.html"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Instantiate the trainer object to drive the model training\n",
|
|
"learning_rate = 0.2\n",
|
|
"lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)\n",
|
|
"learner = C.sgd(z.parameters, lr_schedule)\n",
|
|
"trainer = C.Trainer(z, (loss, label_error), [learner])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"First let us create some helper functions that will be needed to visualize different functions associated with training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define a utility function to compute the moving average sum.\n",
|
|
"# A more efficient implementation is possible with np.cumsum() function\n",
|
|
"def moving_average(a, w=5):\n",
|
|
" if len(a) < w:\n",
|
|
" return a[:] # Need to send a copy of the array\n",
|
|
" return [val if idx < w else sum(a[(idx-w):idx])/w for idx, val in enumerate(a)]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Defines a utility that prints the training progress\n",
|
|
"def print_training_progress(trainer, mb, frequency, verbose=1):\n",
|
|
" training_loss = \"NA\"\n",
|
|
" eval_error = \"NA\"\n",
|
|
"\n",
|
|
" if mb%frequency == 0:\n",
|
|
" training_loss = trainer.previous_minibatch_loss_average\n",
|
|
" eval_error = trainer.previous_minibatch_evaluation_average\n",
|
|
" if verbose: \n",
|
|
" print (\"Minibatch: {0}, Loss: {1:.4f}, Error: {2:.2f}%\".format(mb, training_loss, eval_error*100))\n",
|
|
" \n",
|
|
" return mb, training_loss, eval_error"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='#Run the trainer'></a>\n",
|
|
"### Run the trainer\n",
|
|
"\n",
|
|
"We are now ready to train our fully connected neural net. We want to decide what data we need to feed into the training engine.\n",
|
|
"\n",
|
|
"In this example, each iteration of the optimizer will work on `minibatch_size` sized samples. We would like to train on all 60000 observations. Additionally we will make multiple passes through the data specified by the variable `num_sweeps_to_train_with`. With these parameters we can proceed with training our simple multi-layer perceptron network."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Initialize the parameters for the trainer\n",
|
|
"minibatch_size = 64\n",
|
|
"num_samples_per_sweep = 60000\n",
|
|
"num_sweeps_to_train_with = 10\n",
|
|
"num_minibatches_to_train = (num_samples_per_sweep * num_sweeps_to_train_with) / minibatch_size"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Minibatch: 0, Loss: 2.3106, Error: 81.25%\n",
|
|
"Minibatch: 500, Loss: 0.2747, Error: 7.81%\n",
|
|
"Minibatch: 1000, Loss: 0.0964, Error: 1.56%\n",
|
|
"Minibatch: 1500, Loss: 0.1252, Error: 4.69%\n",
|
|
"Minibatch: 2000, Loss: 0.0086, Error: 0.00%\n",
|
|
"Minibatch: 2500, Loss: 0.0387, Error: 1.56%\n",
|
|
"Minibatch: 3000, Loss: 0.0206, Error: 0.00%\n",
|
|
"Minibatch: 3500, Loss: 0.0486, Error: 3.12%\n",
|
|
"Minibatch: 4000, Loss: 0.0178, Error: 0.00%\n",
|
|
"Minibatch: 4500, Loss: 0.0107, Error: 0.00%\n",
|
|
"Minibatch: 5000, Loss: 0.0077, Error: 0.00%\n",
|
|
"Minibatch: 5500, Loss: 0.0042, Error: 0.00%\n",
|
|
"Minibatch: 6000, Loss: 0.0045, Error: 0.00%\n",
|
|
"Minibatch: 6500, Loss: 0.0292, Error: 0.00%\n",
|
|
"Minibatch: 7000, Loss: 0.0190, Error: 1.56%\n",
|
|
"Minibatch: 7500, Loss: 0.0060, Error: 0.00%\n",
|
|
"Minibatch: 8000, Loss: 0.0031, Error: 0.00%\n",
|
|
"Minibatch: 8500, Loss: 0.0019, Error: 0.00%\n",
|
|
"Minibatch: 9000, Loss: 0.0006, Error: 0.00%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Create the reader to training data set\n",
|
|
"reader_train = create_reader(train_file, True, input_dim, num_output_classes)\n",
|
|
"\n",
|
|
"# Map the data streams to the input and labels.\n",
|
|
"input_map = {\n",
|
|
" label : reader_train.streams.labels,\n",
|
|
" input : reader_train.streams.features\n",
|
|
"} \n",
|
|
"\n",
|
|
"# Run the trainer on and perform model training\n",
|
|
"training_progress_output_freq = 500\n",
|
|
"\n",
|
|
"plotdata = {\"batchsize\":[], \"loss\":[], \"error\":[]}\n",
|
|
"\n",
|
|
"for i in range(0, int(num_minibatches_to_train)):\n",
|
|
" \n",
|
|
" # Read a mini batch from the training data file\n",
|
|
" data = reader_train.next_minibatch(minibatch_size, input_map = input_map)\n",
|
|
" \n",
|
|
" trainer.train_minibatch(data)\n",
|
|
" batchsize, loss, error = print_training_progress(trainer, i, training_progress_output_freq, verbose=1)\n",
|
|
" \n",
|
|
" if not (loss == \"NA\" or error ==\"NA\"):\n",
|
|
" plotdata[\"batchsize\"].append(batchsize)\n",
|
|
" plotdata[\"loss\"].append(loss)\n",
|
|
" plotdata[\"error\"].append(error)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let us plot the errors over the different training minibatches. Note that as we iterate the training loss decreases though we do see some intermediate bumps. \n",
|
|
"\n",
|
|
"Hence, we use smaller minibatches and using `sgd` enables us to have a great scalability while being performant for large data sets. There are advanced variants of the optimizer unique to CNTK that enable harnessing computational efficiency for real world data sets and will be introduced in advanced tutorials. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAACfCAYAAADqDO7LAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHiRJREFUeJzt3XmcFNW5//HPl2ERlM1dZNG4RCFuRBFcwlyVqHhFYzSR\nxF0TNXo1+rvexMRfJGpuzI353WjcolGiKBJi4r4rEOOCqEDYBWURURCURRCQ5fn9cU47RU/PTPdM\n91TPzPN+vfo1XVWnTz1dM9NPnzpV58jMcM455+rSKu0AnHPONQ2eMJxzzuXFE4Zzzrm8eMJwzjmX\nF08Yzjnn8uIJwznnXF48YbRwku6Q9PNCy0oaKGlhaaP7cr/zJB3VGPtqDuLvZmqxy9Yjjn9KOqsU\ndbt0tE47AFcakuYDOwPdzOzTxPpJwAHAbmb2vpldnG+dOcrW6yYeSb2AeUBrM9tcnzqaC0lHAM8Q\njmUroAOwGlBc19vMPiikTjP7B7Bfscs65y2M5ssIH8pDMyskfQ1oTz0/6Iso82Goku9Iqij1PhrC\nzF4xs45m1gnoQzgunTPrspOFolSCdS2eJ4zmbQRwdmL5bOC+ZAFJwyVdF58PlLRQ0pWSlkhaJOmc\nXGWrVulqSUslzZX0vcSGwZImSlopaYGkaxOv+0f8uULSKkmHxtf8QNKMuG6apAMTrzlI0r8kLZf0\nkKS2ud6wpLMlvSLp/0laClwr6VpJIxJleknaLKlVXB4r6br4ulWSnpW0bQ31z5A0OLFcIeljSQdK\naidphKRlMc43JO2Qq546bJEQ4qmd6yS9Rmh99JB0fuJYzZF0fqL80ZLmJZYXSrpC0pQY14OS2hRa\nNm6/WtJHsdwF8Tj2rPMNBb+QNF/SYkn3SuoYt7WP+8kct/GZ4x/f57z4Pt+V9J16HE9XJJ4wmrfx\nQEdJX40fjt8FHqD2b/Y7Ax2BbsAFwG2SOtdSdttY9hzgLkl7xW2rgTPNrDNwAnCRpCFx2zfiz07x\nW/Qbkk4DfgGcEb9tDwE+SezrNOCbwO6EU2rn1PIeDgXeBXYCfhXXZbeqspeHEhLqDkA74D9rqHsk\n8L3E8nHAUjObHF/fCdiVcFwuAtbWEmchziC8507AImAxcHw8Vj8A/hBbkBnZ7+804GjgK8DBwJmF\nlpX078AlwEBgb+CoHK+tyQ8Ix+0bwB6E4/P7uO1cQsu3W1z/I2BdTCi/A46O7/NwYEqe+3Ml4Amj\n+cu0MgYBM4EP6yj/BXC9mW0ys2cIH/xfraGsAf/XzDaY2cvAU8B3AMzsZTObHp9PA0YRPmiSkonr\nfOB/zGxifM1cM0t2qt9sZkvMbAXwBJBsfWRbZGa3m9lmM1tfx/vNGG5m78Xyo2up/yFgiKSt4vLQ\nuA5gA7AdsLcFk8xsdZ77r8u9ZjY7/l42mdlTZrYAwMzGAS8BR9by+v81s6Vmthx4ktqPX01lTwPu\niXGsBX5ZQPzfA26K/WZrgJ9RlXg3ANtTddwmmtnncdtmYD9J7eLvf1YB+3RF5gmj+XuA8I95DnB/\nHuU/yeqI/hzYpoayy81sXWJ5AeFbIpIOlTQmnq5ZAVxI+FCoSQ/gvVq2L8kzJoD6XL21OJ/6zew9\nYAZwoqT2hJbQyLh5BPAcMErSB5JuVPH6ULZ4T5L+PZ66+UTScsIXgtqObyHHr6ay3bLiWEj+/VDd\nCH8fGQuAdvGU3Z+BF4HR8VTXf0tqZWafERLypcBiSY8nWrAuBZ4wmjkze5/Q+X088PciV981fmhm\n9KSqBfMg8Ciwq5l1Af5I1YdLrtMYCwmnKoohu/41hKuPMnZpYP2jCEn4JGC6mc0FMLONZna9mfUB\nDgNOBIp1WemX7ym2bv5KON22g5l1BV6g9BcRfAR0Tyz3JP9TUh8CvRLLvYD1sSWzwcyuM7PewBHA\nKcD3AczsOTMbRDj9+R7h78ilxBNGy3AecFQ8jVBMAn4pqY2kIwl9FaPjtm0ILZANkvqx5Xn/pYRT\nDckE8SfgPyX1BZC0h6QeRYpzMvANST1if8xPG1jfKEJ/ysVUtS6QVCnpa7G/aDXhVEuhlw3n86Hf\nDmgDLAMs9i0cXeB+6mM0cL6kvSV1AK4p4LUPAVfGCw46AjcQj52kf5PUR5JIHDdJO8eWVHtgIyHx\nbyrmG3KF8YTRfH35zc/M5mX6BrK3FVJPDh8BywnfHkcAF5rZnLjtR8D1klYSPlj+kohnLeHb8auS\nPpXUz8wejutGSloFPELoAC003upvwOzFuP8pwJuEPpAtihRY32LgdaA/ifdF+Bb8MLASmA6MJRyX\nzE2Pt+dTfV3rzGwlcAWhBfcJ4Rt59nuqq86Cy5rZk8AdwMvAO8ArcVNN/UTJuu4mHKt/Ei5IWAn8\nOG7rRmj9rgSmAs8TkkkFcBXh72spMIDQ6e5SolJOoCSpO+G8+U6Eb1p3m9ktWWUGAo8Bc+Oqv5vZ\nDSULyjlXFPGqrLfNrF3asbjGUeo7vTcCV5rZZEnbAG9Lej7HlQ4vm9mQHK93zpURSScTrobrCNxI\naOW4FqKkp6TMbHG8Pp14eeFMwjXq2fzOVeeahksIfSezCVdQXZpuOK4xNdpYUpJ2I1zP/UaOzQMk\nTSbckHSVmc1orLicc/mLVyy5FqpREkY8HfUwcHmOG5neBnqa2eeSjic0cffOUUfa4x8551yTZGZF\nOYtT8qukJLUmJIsRZvZY9nYzW525qzPeWdxGNYzjY2Zl9bj22mtTj6GpxOUxeUwtIa5yjKmYGuOy\n2nuBGWZ2c66NknZKPO9HuHLr01xlnXPOpaekp6QkHU64Y3OqwjwMRhhDphdgZnYXcKqkiwk366wl\nDJDnnHOuzJQ0YZjZq4Sbb2orcxtwWynjKJXKysq0Q8ipHOPymPLjMeWvHOMqx5iKqaQ37hWTJGsq\nsTrnXLmQhDWVTm/nnHPNgycM55xzefGE4ZxzLi9NKmGsWpV2BM4513I1qYQxYULaETjnXMvVpBLG\na6+lHYFzzrVcTSphvP562hE451zL1aQSxqJF4LdiOOdcOvzGPeeca8b8xj3nnHONrqQJQ1J3SWMk\nTZc0VdJlNZS7RdIcSZMlHVjKmJxzztVP6nN6x0mT9jCzvSQdCtwJ9C9xXM455wpUDnN6nwTcH8u8\nAXROzpHhnHOuPDRaH0Ytc3rvCixMLC+ielL50uzZMH9+kYNzzjlXp0ZJGHXM6V2QBx6Au+8uTlzO\nOefyV+o+jDrn9Ca0KHoklrvHddUMGzaM994Ld3wPGlTZ7Ccrcc65Qo0bN45x48aVpO6S34ch6X5g\nmZldWcP2wcAlZnaCpP7A782sWqd35j6M5cuhZ09YvhxalzzdOedc01bM+zBSn9PbzJ6WNFjSu8Aa\n4Nza6uzaNSSMKVOgb99SRu+ccy4p9Tm9Y7lLC6l3wIAwrpQnDOecazxN8qTOqafC6gZ1nTvnnCuU\njyXlnHPNmI8l5ZxzrtF5wnDOOZcXTxjOOefy4gnDOedcXppswjCDq66C9evTjsQ551qGJpswJHjp\nJZg0Ke1InHOuZWiyCQPCDXyvvZZ2FM451zI06YRx2GHhjm/nnHOl1+QTxmuvhf4M55xzpdWkE8Zu\nu8GmTfD++2lH4pxzzV9JE4akeyQtkTSlhu0DJa2QNDE+rimsfhg1Crp0KU68zjnnalbSsaQkHQGs\nBu43s/1zbB8I/B8zG5JHXT6WlHPOFajJjCVlZq8Ay+soVpQ34pxzrrTKoQ9jgKTJkp6S1DvtYJxz\nzuWW9nwYbwM9zexzSccDjwJ711R42LBhXz6vrPQ5vZ1zLltTn9O7F/BErj6MHGXnAV83s09zbPM+\nDOecK1Cj92FI2kNSu/i8UtJlkvK9NknU0E8haafE836EBFYtWdTl5ZfhrLMKfZVzzrlC5NXCkDQZ\nOBjYDXgaeAzoY2aD63jdSKAS2A5YAlwLtAXMzO6SdAlwMbABWAtcYWZv1FBXjS2MDz6Agw6Cjz8O\nl9o655wLitnCyDdhTDSzvpKuAtaZ2R8kTTKzg4oRRD7qOiXVqxe8+CLstVdjReScc+UvjctqN0ga\nCpwNPBnXtSlGAMXiAxE651xp5ZswzgUGAL8ys3mSdgdGlC6swvlAhM45V1oFXyUlqSvQw8xyDvdR\nKnWdknrrLbjoovDTOedckEYfxjhgCOG+jbeBj4FXzezKYgSRj7oSxqZN4dG2bWNF5Jxz5S+NPozO\nZrYKOIUwLtShwDHFCKBYKio8WTjnXCnlmzBaS9oF+A5Vnd7OOedakHwTxnXAc8B7ZvampK8Ac0oX\nlnPOuXJT8qFBisWHBnHOucKlMTRId0mPSPo4Pv4mqXsxAii2jz+GFSvSjsI555qffE9JDQceB7rF\nxxNxXdm56ioYPTrtKJxzrvnJN2HsYGbDzWxjfPwZ2KGEcdXbgAF+A59zzpVCvgnjE0lnSKqIjzOA\nT+p6UV1zescyt0iaEydROjDfwGty2GE+RIhzzpVCvgnjPMIltYuBj4BTgXPyeN1w4NiaNsZJk/Yw\ns72AC4E784ynRn36wOLFsGxZQ2tyzjmXlFfCMLMFZjbEzHYwsx3N7GTg23m8rq45vU8C7o9l3wA6\nJ+fIqI+KCujXD8aPb0gtzjnnsjVkTu9iDAuyK7AwsbwormuQU06B9esbWotzzrmkhszp3ehTFeU7\np/fFFzdOPM45V27Kck5vSe+bWc88ytU4p7ekO4GxZvaXuDwLGGhmS3KU9Rv3nHOuQMW8ca/WFoak\nz4Bcn9IC2ue5jxrn9Cbc23EJ8BdJ/YEVuZKFc8659NWaMMysY0MqT87pLel9sub0NrOnJQ2W9C6w\nhjBRk3POuTLkY0k551wzlsZ8GE3O+vVw661pR+Gcc81Hs21hmMH228PUqdCtWwkDc865MuYtjDxI\nPq6Uc84VU7NNGODjSjnnXDF5wnDOOZeXZtuHAbBmDey4I3zyCWy1VYkCc865MuZ9GHnaeutwpdSG\nDWlH4pxzTV+zbmE451xL5y0M55xzjc4ThnPOubx4wnDOOZeXkicMScdJmiVptqSf5Ng+UNIKSRPj\n45pSx+Scc65wJU0YkloBtxLm9e4DDJW0T46iL5tZ3/i4odhxPPss/PrXxa61eXvvPRg5Mu0onHPl\npNQtjH7AnDgn+AZgFGEe72wlnb2vUyd4+OFS7qF5Wb4cBg+GK66AG29MOxrnXLkodcLInrP7A3LP\n2T1A0mRJT0nqXewg+vaFWbPCjXyudhs3wumnw/HHw6RJYSwuP27OOSiPTu+3gZ5mdiDh9NWjxd7B\nVlvB/vvDm28Wu+bmZ+xYaNUKbropjPL72GPhBkjnnKt1xr0iWAQk5/3uHtd9ycxWJ54/I+l2Sdua\n2afZlQ0bNuzL55WVlVRWVuYdSGZcqQJe0iINGgRHHQUVFWlH4pyrj3HjxjFu3LiS1F3SO70lVQDv\nAEcDHwETgKFmNjNRZqfMPN6S+gGjzWy3HHU16E7vv/0Nhg+HJ5+sdxXOOdfkFPNO75K2MMxsk6RL\ngecJp7/uMbOZki4kzusNnCrpYmADsBb4biliOe44OOKIUtTcspjBU0/BCSeEOUeccy2HjyXlCrJq\nFRx+OAwZAjfc4EnDuXLnY0m5oti8Gc4/H2bMyP81nTrBmDHh1N7PfhZaHM65lqHUnd6ujF1zDcyZ\nA3vuWdjrdtgBXnoJjjkmJJ0bb/SWhnMtgSeMFmrkSHjoIZgwAdq2Lfz1228fksagQbDTTnDllcWP\n0TlXXlpcH8YXX4Rvw23aFCGoJmrChNBpPWYM7Ldfw+r69NNws9+OOxYnNudccRWzD6PFJYzBg6Fj\nR7j8chgwoOWdSlm3DvbZB265JXRcO+eaN08YDbB0abgf4557ws1p558PZ57Zsr4hL1gAvXqlHYVz\nrjF4wigCM3jlFfjTn2DxYnjuuaJV3eJt3hxabi2t9eZcOfKEUWRm/uFWTL/9LcybB7feGsalcs6l\nxxNGI7nrLujSBU46Cdq1a9RdN2mrVoU76/ffH26/3ZOGc2nyG/cayY47wh//CN27h7khpk1LO6LC\n/etfjX+6rVOnsM9p0+Cii8IpKudc0+cJoxYnnxzuNRg/Hjp0gGOPDcNibNhQ++s2b4YVK2D+/HBq\nJpdp0+Ccc8I+Kivh1FPhttvCjXTF8vHHoXW0YkXx6sxXx47wzDMwcyb88IeeNJxrDvyUVAE2boS3\n3oL+/atvGz8ehg4Ns9V99hlss004nXXMMeGKrGwffAAvvABdu0LnzmF57FjYdVe4/vqGx7p+fdh3\nZWVx6quv1atDK2PEiOr9ROvWwahRoQXXvTv06OFzbzhXbMU8JYWZlfQBHAfMAmYDP6mhzC3AHGAy\ncGANZazcjB079svna9aYzZ1r9umnZhs3Fn9fTz5pds89YR91GTNmrJ13ntm3vmW2aVPxY6mP5LHK\nWLbM7MwzzSorzfbc02yrrcy6dDEbNCh3HRs2mK1YYbZ2bXGOca6Y0uYx5a8c4yrHmOJnZ1E+z0t6\nSkpSK8IsescCfYChkvbJKnM8sIeZ7QVcCNxZypiKKTlJSYcOsPvuocVQismHKipCi2TAANhtNzj3\nXLj//nCndbbf/GYcb70VtpdLh3OuCV222y7EOHZsOBX3+efh56235q5j9uzQCuncOdypX1ERZlPM\n1eKDUNfXvx62H3kkHH106Iz/j//IHdOyZXDHHaE19Mgj4Xi//nrYb2Mp1cQ3DVGOMUF5xlWOMRVT\nqceS6gfMMbMFAJJGAScRWhwZJwH3A5jZG5I6JydVcsFxx4WHWZiffOxYeOIJ6N0btt12y7K77BI6\n67fZJp1Y60sKY1Rtv33u7b17hyuwMjZtCkO91NQ/suuu4Ths2BAeX3wRfnbokLv82rXhIoHVq7d8\n9OwJo0dXLz91Kpx2WjiNlkzM++wTkk62mTPh+9+vvn7ffeHBB6uvf+cduPDCkBhbt676uffeYQrd\nbAsWhIEgM2VbtQqPnj3hssuql//ww3C6tFWrcOwz5XfZJdzMmu2TT8JcKG3abPnYdtvcSXvdOli0\naMuymbg6d65e/osvQr+bWXhs3hx+tm0bfpfZNmyA6dOrymce7duHY5RtzZqq8lBVfuutwxV92Vav\nzn2hy9Zb5x5SZ/Xq8H4nTKhevk+f3PHMnFl9/dZbh7+JXOXfeadqOXOKt0MH+OpXq5f//HN4993q\n6xui1AljV2BhYvkDQhKprcyiuM4TRg5S+GPad1/40Y9yl+nVq2XcyV1RET4catKhAxx8cP719egB\ndxbQvt1rL3j00fBBkexeqykh9ewZbhTNVtN76NYNhg0LfWebNlX97Ngxd/n27cMH38aN4bF5c3h0\n6ZK7vFlVws18OG/eXPNglKtWhVZXJgFnHnvskTthzJ4dLrpIljWDvn3DxSTZZsyompgrk8QkOOCA\nMLd8tuXLQ8LOlMs8+vQJA2tmW7AALrlky5tKpfBF5N57q5d//3348Y+rr9933zBaRK7yTz8dvnRk\nl7/vvurl588P/Xu56s/1hWP+fLjggvA8+ffWu3fuLxxz58IZZ1Rf3xClnqL128CxZvbDuHwG0M/M\nLkuUeQL4tZm9FpdfBP7LzCZm1dU0euedc67MWFOYopXQWuiZWO4e12WX6VFHmeL18jvnnKuXUneJ\nvgnsKamXpLbA6cDjWWUeB84CkNQfWOH9F845V35K2sIws02SLgWeJySne8xspqQLw2a7y8yeljRY\n0rvAGuDcUsbknHOufprMjXvOOefSVSZX6ddO0nGSZkmaLeknJd7XPZKWSJqSWNdV0vOS3pH0nKTO\niW1XS5ojaaakbybW95U0Jcb8+wbG1F3SGEnTJU2VdFnacUlqJ+kNSZNiTNemHVOivlaSJkp6vBxi\nkjRf0r/isZpQJjF1lvTXuI/pkg4tg5j2jsdoYvy5UtJlZRDXFZKmxfoelNS2DGK6PP7fNe7nQbHu\nACzVg5DU3gV6AW0Id4PvU8L9HQEcCExJrPsN4cotgJ8AN8bnvYFJhFN7u8U4M622N4BD4vOnCVeL\n1TemnYl3wAPbAO8A+5RBXB3izwpgPOGS6VRjinVcATwAPF4mv7+5QNesdWnH9Gfg3Pi8NdA57Ziy\n4msFfEi4ICa1uIBu8ffXNi7/BTg75Zj6AFOAdoT/veeBPRojpgb/Ykv9APoDzySWf0oNQ4wUcZ+9\n2DJhzAJ2is93BmbligV4Bjg0lpmRWH86cEcR43sUOKZc4gI6AG8Bh6QdE+EquxeASqoSRtoxzQO2\ny1qXWkxAJ+C9HOvL4u8p1vVN4J9px0VIGAuAroQP3MfT/t8DTgXuTixfA1wFzCx1TE3hlFSum/9y\n3PdZUjtavHLLzBYDmQlda7rpcFdCnBlFi1nSboQW0HjCH0dqccVTP5OAxcALZvZm2jEB/0v450l2\nzqUdkwEvSHpT0gVlENPuwDJJw+Ppn7skdUg5pmzfBUbG56nFZWYfAr8D3o/1rzSzF9OMCZgGHBlP\nQXUABhNaYiWPqSkkjHKUypUCkrYBHgYuN7PVOeJo1LjMbLOZHUT4Vt9PUp80Y5J0ArDEzCYDtd23\n09i/v8PNrC/hH/sSSUfmiKExY2oN9AVui3GtIXwLTfXvKUNSG2AI8Nca4mjMv6kuhOGLehFaG1tL\n+n6aMZnZLMLppxcIp5EmAZtyFS32vptCwsjn5r9SWyJpJwBJOwMfJ2LLddNhXjcjFkJSa0KyGGFm\nmYESUo8LwMxWAeMIIxOnGdPhwBBJc4GHgKMkjQAWp3mczOyj+HMp4XRiP9I9Th8AC83srbj8N0IC\nKYu/J+B44G0zWxaX04zrGGCumX1qZpuAR4DDUo4JMxtuZgebWSWwgtCvWfKYmkLCyOfmv2ITW35D\nfRw4Jz4/G3gssf70eNXE7sCewITYHFwpqZ8kEW5MzDEaTkHuJZxvvLkc4pK0feYqDEntgUGEc6ip\nxWRmPzOznmb2FcLfyRgzOxN4Iq2YJHWILUMkbU04Nz+VdI/TEmChpMwQfUcD09OMKctQQsLPSDOu\n94H+kraKdR0NzEg5JiTtEH/2BL5FOH1X+piK0UFV6gfhm+s7hDkzflrifY0kXJ2xnvDHci6hw+vF\nGMPzQJdE+asJVx3MBL6ZWP91wgfDHODmBsZ0OKHJOZnQ/JwYj8m2acUF7BfjmEy4YuPncX1qMWXF\nN5CqTu80j9Puid/b1Mzfb9rHCTiA8GVsMvB3wlVSqf/uCBdQLAU6JtalfayujfVPAe4jXK2Zdkwv\nE/oyJgGVjXWc/MY955xzeWkKp6Scc86VAU8Yzjnn8uIJwznnXF48YTjnnMuLJwznnHN58YThnHMu\nL54wXNmRtFnS/YnlCklLVTVc+YmS/quOOnaRNDo+P1vSHwqM4eo8ygyXdEoh9RaTpLGS+qa1f9fy\neMJw5WgN8DVJ7eLyIBKDp5nZE2b2P7VVYGYfmdl3kqsKjOFnBZZvUiRVpB2Da3o8Ybhy9TRwQny+\nxVARyRZD/JZ/s6RXJb2b+cYfh5KZmqivZ/xG/o6kXyTqeiSOIjs1M5KspF8D7eNIriPiurNUNQnS\nfYl6B2bvOynGMUNhRNhpkp7NJMJkC0HSdpLmJd7fIwqT4cyVdInCJD4TJb2mMCBexlkxpimSDomv\n76AwEdh4SW9LOjFR72OSXiLcEexcQTxhuHJkwChgaPxw3Z8w0Ut2mYydzexw4ETCKJ65yhxCGHPn\nAOC0xKmcc83skLj9ckldzexq4HMz62tmZ0rqTWhxVFoYnffyPPadtCfwBzP7GrAS+HYt7zujD3Ay\nYaDCXwGrLYwsO54w5k9G+xjTJYTxxgB+DrxkZv2Bo4Cb4nhfAAcBp5jZv9UQg3M18oThypKZTSPM\nDjYUeIrahyt/NL5mJlVzAGR7wcxWmNk6wthJR8T1P5Y0mfBB3B3YK65P7u8o4K9mtjzuZ0WB+55n\nZpnWztvxfdVlrJl9bmHE1hXAk3H91KzXPxT3/0+go6ROhAEOf6owV8k4oC1VIz6/YGYr89i/c9W0\nTjsA52rxOPBbwux529dSbn3ieU2Jpdr8BZIGEpLBoWa2XtJYYKsCY8xn38kymxL72EjVl7bs/SZf\nY4nlzWz5f5trXgYB3zazOckNkvoT+oecqxdvYbhylPngvRf4pZlNr8drsw2S1CWemjkZeJUwQuvy\nmCz2IUwHnPFFomN4DOE01rYAkroWuO+a1s8HDo7PT6uhTF2+G2M6gjAb3GfAc8BlX+5cOrCedTu3\nBU8YrhwZgJktMrNb8ylby3LGBMKpqMmE00sTgWeBNpKmA/8NvJ4ofxcwVdIIM5sRt/8jnub5XYH7\nrmn9TcDFkt4mDE1dk9rqXSdpInA7cF5cfz3hfU2RNA24rpa6ncubD2/unHMuL97CcM45lxdPGM45\n5/LiCcM551xePGE455zLiycM55xzefGE4ZxzLi+eMJxzzuXl/wPDVUMFvB4I7wAAAABJRU5ErkJg\ngg==\n",
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x1ae8b1cb0f0>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAACfCAYAAADqDO7LAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXm4FMXV/z9fERBEEJe4EEHFLeKKirhfNRqMUYzGBaK4\nJC7RvBjFRI3+FKP+FJcoaoyaKBpNMBqN4r5EUF8VNyAgghgBF0QUF4SAeIXz/lE13L7D3Jmee2e7\ncj7PM89MV1efOtPT06er6pxTMjMcx3EcpxArVVsBx3Ecp3XgBsNxHMdJhRsMx3EcJxVuMBzHcZxU\nuMFwHMdxUuEGw3Ecx0mFG4waRdIfJZ1XbF1Je0l6v7zaLWt3hqR9KtFWa0PSUkkbV/rYlpJsu5hr\nMIec+ZI2LKVuTvVxg1FhJM2U9JWkNbLKx8c/a3cAM/uFmV2aRmaOus0KrpHUI+rg1wUtPh8tCXBq\n8lhJYyQtkvSlpI8l3SdpnRa01WTbaa9BSaMlndBIiNlqZjazhHpl2popaWH8/vPj+3WlbsfJjd8Y\nKo8BM4ABmQJJWwEdaNlNphQo6qCyNyS1KXcbJaK5v0lLzmG+Yw041cw6A5sBqwPX5BTSPENX9t++\nhRhwoJl1jkaps5kNzlUx1zVW7HXXiq7TiuAGozrcCRyb2D4WuCNZQdIISb+Ln/eS9L6kMyXNkTRL\n0nG56jYU6VxJn0iaLmlgYscPJY2TNE/Su5IuTBz3bHz/Ij657RyPOVHSm7HsDUnbJY7ZXtK/JX0u\naaSkdrm+sKRjJf2vpN9L+gS4UNKFku5M1Gn0RB+fXH8Xj/tS0uPZPbPEsW9K+mFiu018At9OUntJ\nd0qaG/V8WdLaueSkRdJOkl6M8mZJul7SylnVDpT0TtTjiqzjT4g6fyrpsUzPMm3zAGb2BXAfsFWU\nOULSjZIekTQfqJPUTtJV8beeHfe3T+jxa0kfSvpA0vEkDGT2dSWpf+wJz5P0tqT9JV0C7AHckHza\nV+Ohrc6S/hLPwwwlhrnidfG8pCslfRbPV78033+5wsbX2FzCNZarTJLOV+itfCTpdkmdo4zMNXiC\npHeBf6X9UVYE3GBUh7HAapI2jzfHI4G7yP90ty6wGrA+8HPgD5K65Km7Rqx7HHCLpE3jvgXAMWbW\nBTgQOEXSwXHfnvG9c3xye1nS4cAFwNHxqfZg4NNEW4cD+wMbAdvG9ppiZ+A/wDpAZqgj+wk+e3sA\nwaCuDbQHzmpC9t+AgYntfsAnZjYhHt8Z6EY4L6cAi/LomYYlwK+ivF2AfYBTs+ocAvSOr/6KwzaS\n+gPnxP1rA88DI4tVQNJawGHAuETxAOBiM1sNeAEYBmwCbBPfuxF+T+KN+UxgX2BT4Pt52upDeKgZ\nEq+dPYGZZnZ+1P+XWU/7yd/xBsK1uyFQBwyKxilDH2AKsCZwJXBrMechi8w19h0arrHssuOBQcBe\nwMZRtxuy5OwJbAH8oAW6fOtwg1E9Mr2M/Qh/lg8L1P+acCNYYmaPEW78mzdR14D/Z2b1ZvYc8Ahw\nBICZPWdmk+PnN4C7CX+cJEnD9TPgCjMbF4+ZbmbJSfXhZjYnPu0+BCR7H9nMMrMbzWypmS0u8H0z\njDCzd2L9e/LIHwkcLGmVuD2AhptwPeFmtJkFxpvZgpTt58TMxpnZK1Hee8AtLH8eLzezeWb2AXAt\nDcOQJwOXmdk0M1sKXA5sJ2mDlM1fL+kzYDzhuhmS2PegmY2NOi4GTgTOiHr8N7aV0eNwwvmdYmaL\ngKF52jwBuNXMnomyZ5vZtDz1BcuGxY4EzjGzhWb2LnA1cEyi7rtmdpuFxHZ3AOtK+k4e2Q/E3sjn\n8f1niX25rrHssoHA783sXTNbCJwLHKWGITwDLjSzRUVcpysE2V1op3LcBTxHeDL/S4r6n8abS4aF\nQKcm6n5uZl8ltt8l9DZQGGa6jDCM0S6+7s3T7gbAO3n2z8nSab08dZvjvfVRlvyc39nM3pH0JnCQ\npIcJPaEL4u47ge8Cd8de2V3AeWa2pBn6ABB7bL8HdiTMP60MvJ5V7YPE52W/AdADGC7p6ow4wk2q\nG+nO0f+Y2W1N7Ft2fBx26wi8Li17BliJhgeC9YHXsnRsqpe7AeHBo1jWIpyb97La6ZbYXvYbm9ki\nBWU7AR83IbO/mY1uYl+u85ddtn7UIanPyoSeb4YPcJbDexhVIj6VzgAOAO4vsfiukjoktrvT0IP5\nK/AA0M3MVgdupuEmkWuC932gZ4n0ypb/X8INLUM+Y5OGuwlPj/2ByWY2HcDMvjGzi82sF7ArcBBh\nSKIl/JHQM+wZz+N5LH+zTfYYetDwG7wPnGxma8RXVzPrlOkZtJDkOZ5LMLK9Em2tHoeUAGbn0LGp\nSf5810E+x4C5hB5ej6x2ZuU5phCFnAIKlX2YQ596Gj/8VNsBpSZxg1FdTgD2icMBpUTARZLaStqD\nMFdxT9zXidADqY/j0slx/0+ApTS+MfwZOEtSbwBJPYsYOinEBGBPSRvEJ/9zWijvbsJ8yi8IcxoA\nSKqTtFUcclhAuDkszS1iOQSsEifOMy8Rxr2/NLOFkraIbWbza0mrx/M1OOoHcBPwW0lbRv26SPpJ\n8V83P3GI50/AtbG3gaRukvaPVe4BjpP0PUkdaeiR5eJW4HhJe8dJ4/UlZYZE5xDmAnLpsDS2c6mk\nTpJ6AGcQen3VYiRwhqQNJXUizGvcnejB17qnWNVwg1F5kn7uMzJzA9n7ipGTg9nA54QnqTsJT7Nv\nx32nAhdLmgecD/w9oc8iwp/nhTg23MfM/hHL/ibpS+CfhIneYvVd/guYPR3bnwi8SpgDaVSlSHkf\nAS8BfUl8L4ITwD+AecBkYDTxhqUQnHZjPrHAfMKT+qL4vjdh3uCn8ZzcTIMxSB73IGGYalz8brdF\nPR8gzCXcLekLwvfvl3VsPn2K2Xc2YcJ3bGzrSYI7Lmb2OGFu5RlgGnk8gszsVcJk8bWE8ziG0HMF\nGA4cruDxdW0OXQYTztt0wjDsXWY2osjvkeSh6JGVed1XoH42txF+/+cIw60Lo45p219hUbkXUIqe\nGNcSjNOtZjYsa//qhB+wJ+EPeYKZvVlWpRzHcZyiKWsPIw4B3EBwTesFDIjd9yS/Bcab2bYEryGP\n2nQcx6lByj0k1Qd4O7qv1RO67f2z6mxJ6BJjZm8BG6qFQVWO4zhO6clrMBSiZZtyX0tDtpvgBzR2\npwP4N3BobK8PYVz0uy1o03EcxykDeeMwzGxJDJPvYmbzyqTD5QSf9HHAJEIw0nL+8ZJ8IspxHKcZ\nmFlJPL/SDEktACZJulXSdZlXSvmzaPCkgNBzaOR/bWbzzewEM+ttZscSwven5xJmZjX1uvDCC6uu\nQ2vRy3VynVYEvWpRp1KSJtL7fpofWPYqsEn0vZ4NHEUiSysEH3RgoYW4gBOBZ62FaRscx3Gc0lPQ\nYJjZHQoZSDeLRW9ZmMAuiIUhrV8SfL8zbrVTJJ0cdtstwPeAOyQtJfjI/6xpiY7jOE61KGgwJNUR\nEoLNJERAbiDpWAtJ7QpiITho86yymxOfx2bvby3U1dVVW4Wc1KJerlM6XKf01KJetahTKSkYuCfp\ndWCgBZdXJG0GjDSzHSqgX1IPK/V4nOM4zrcdSVgFJ73bZowFgIWUxm1L0bjjOI7TekhjMF6T9OeY\nwK1O0p9onBI5L5L6SZoqaZqks3Ps7yxplKQJkiYpsZKc4ziOUzukGZJqD5wG7B6LngdutBQLi8TU\nINMIK3p9SPCaOsrMpibqnEtY4e3cuILYW8A6ZvZNliwfknIcxymSUg5J5Z30VlgA/TYz+ylhsZhi\nWZYaJMrLpAaZmqhjhFTRxPdPs42F4ziOU33yDklZWJGsR3SrbQ5pUoPcAGwp6UNCmpDTm9mW4ziO\nU0bSBO5NJ6yPMIqwQhoAZtacHkcufkDIVruPpJ7AU5K2yRm899VXsMoqyxU7juM45SeNwXgnvlai\nYegoLQVTgxAWZbkMlq3LPAPYghwT60NPOw02CIu91dXVfet9nh3HcYplzJgxjBkzpiyy8056xzmM\nYWZ2VrOEh+PfIkx6zwZeAQaY2ZREnT8AH5vZRZLWIRiKbc3ssyxZZldeCWc1SxXHcZwVkopNesfU\nHrs1V3jK1CCXALdLmhgP+022sVjGSy81VxXHcRynhaRxq/0jYaL6XhrPYTQ3IWGzkGS23nowaxbI\n12h3HMdJQ8V6GJFVgE+BfRJlRvMz2DafN95wY+E4jlMlCvYwagUP3HMcxymeiuSSknRP4vOwrH1P\nlqJxx3Ecp/WQL3Bv08Tn/bL2rZ22gRS5pM6SNF7SuJhL6htJq6eV7ziO41SGfAYj3/hPqrGhmEvq\nBkJwXi9ggKQtGgkyu8rMtjez3sC5wBgz+yKNfMdxHKdy5Jv07ihpe4JR6RA/K746pJSfJpdUkgHA\nyLwSFy2C+nro3DmlCo7jOE4pyGcwZtOQcPAjGicf/Cil/Fy5pPrkqiipA9CPkBm3ac4/H9ZaC849\nN6UKjuM4Tilo0mCY2d6VVAQ4CPjffMNRQ4cOhfffh4ceom6XXTw1iOM4ThZVSw3SYuFSX2ComfWL\n2+cQIryH5ah7P3CPmd3dhKzgVjtrFmy7LXzyicdkOI7jFKDSS7S2hFeBTSRlUqQfBYzKriSpC7AX\n8GBBid26QceO8PbbpdbVcRzHyUNZDUZcTyOTS2oycHcml5SkkxJVDwGeMLNFqQTvuqvnlXIcx6kw\nqYakJHUDepCY8zCz58qoVy4dGiK9R4wI3lKnnlpJFRzHcVodpRySSpN8cBhwJPAmsCQWm5kdXAoF\n0uKpQRzHcYqn0gbjLWAbM1tcigabixsMx3Gc4qn0pPd0oG1zGyiUGiTWqYvpQd6QNLq5bTmO4zjl\nI00P4z5gW+BfwLJehpkNLig8pAaZRlhx70OC19RRZjY1UacL8CKwv5nNkrSWmc3NIct7GI7jOEVS\n6fUwRpHDFTYlaVKDDATuM7NZALmMheM4jlN9ChoMM7sjxlBsFoveMrP6lPLTpAbZDGgbh6I6AdeZ\n2Z0FJZvB7bfDscfCSuUOJ3Ecx3EKGgxJdcAdwExC4sENJB1bQrfalYHehBX9VgVekvSSmf0nu+LQ\noUOXfa6rq6Pu0kuhTx/o1atEqjiO47RuqpoaRNLrwEAzeytubwaMNLMdCgpPkRokToSvYmYXxe0/\nA4+Z2X1Zspafwxg0CPbYA048seAXdRzHWRGptJdU24yxADCzaaT3mkqTGuRBYHdJbSR1BHYGpqSS\nvssuHvHtOI5TIdJMer8Wn/rvits/BV5LI9zMlkjKpAZZCbg1kxok7LZbzGyqpCeAiYTAwFvM7M1U\n2u+6Kwwfnqqq4ziO0zLSDEm1J6xRsXsseh64sdKBfDmHpJYsga5dYcYMWHPNSqrjOI7TKqhopHet\n0GQcxs03Q//+sO66lVfKcRynxqmIwZB0j5kdIWkSOdbwNrNtSqFAWjxwz3Ecp3gqZTDWM7PZknrk\n2p8JxivYgNQPuJaGOYxhWfsz62BMj0X3m9klOeS4wXAcxymSikR6m9ns+PFUM2uUAypmsM2ZFyqr\n3krADSRSg0h6MJkaJPJcpbPfOo7jOMWRxq12vxxlB6SUvyw1SIwOz6QGycbXWnUcx6lxmjQYkn4R\n5y+2kDQx8ZoBTEopP1dqkG456u0iaYKkRyRtmVp7x3Ecp2Lki8P4G/AYcBlwTqJ8vpl9VkIdXge6\nm9lCSQcAD9CQt6oRy6UGqasLG6++Ck8+CeedV0K1HMdxWh/VTg3SF5hsZvPjdmfge2b2ckHhKVKD\n5DhmBrBDtlHKO+k9ZQr86EfwzjuFVHIcx1mhqHRqkD8CCxLbC2JZGgqmBpG0TuJzH4IRK64Hs/nm\n8PnnMGdOUYc5juM46UljMBo92pvZUtKlFMHMlgCZ1CCTgbszqUEknRSr/SSutDee4H57ZFHfAEJ6\n8759Pa+U4zhOGUkzJHU/MIaGXsWpwN5mdkh5VVtOj/xxGBdfDPPnwxVXVE4px3GcGqfSQ1KnALsC\nswheTjsDJ+U9ohp45lrHcZyy0vpzSWVYtAhmzoTvfa9iOjmO49Q6lUoN8hszu0LS9eTOJTW4FAqk\nxVODOI7jFE+lhqQyixi9RoiVyH6lQlI/SVMlTYur6zVVbydJ9ZIOTSvbcRzHqRxlHZKKuaSmkcgl\nBRyVnUsq1nsKWATcZmb355DlPQzHcZwiqUjyQUkPkWMoKkPKZIHLcklFmZlcUtnJB/8H+AewUwqZ\njuM4ThXIF09xVXw/FFiXhiVaBwBpI+Ry5ZLqk6wgaX3gEDPbOwbutQwzWLoU2rRpsSjHcRyngXzp\nzZ8FkHS1me2Y2PWQpFRreqfkWhqnSm+y69RkLqkkgwbBQQfBEUeUTEHHcZzWQrVzSU0BDjSz6XF7\nI+BRMyvov5oml5SkzMJJAtYC/gucZGbZKUTSzWEMGwazZ8O11xau6ziO8y2nInMYCc4AxsQbu4Ae\nwMkp5S/LJQXMJuSSGpCsYGYbZz5LGgE8lG0simLXXWHIkGYf7jiO4+SmoMEws8clbQpsEYummtni\nNMLNbImkTC6pzBKtUySdHHbbLdmHFKF7bnbcESZPDoF8HTq0WJzjOI4TSDMk1RE4E+hhZidG47G5\nmT1cCQUTeqR3q+3TB66+GvbYo7xKOY7j1DiVziU1Avga2CVuzwIuKUXjZWPPPX1tDMdxnBKTpofx\nmpntKGm8mW0fy/5tZttWRMMGPdL3MMxAvky44zhOpXsYX0vqQJxfkNQTSDWHEevnTQ0i6WBJ/5Y0\nXtIrknZLrX3TjbZYhOM4jtOYND2M/YDzgS0Jk9e7AceZ2ZiCwlOkBpHU0cwWxs9bA/fkctn11CCO\n4zjFUzG3WkkipPE4FOhLcKs93czmppRfMDVIxlhEOgFLU2vvOI7jVIy8BsPMTNKjZrY18Egz5BdM\nDQIg6RDgMmBt4MBmtOM4juOUmTSBe+Mk7WRmr5ZLCTN7AHhA0u4ED6z9ctVLlRokQ309jB3rrrWO\n46xQVDs1yFRgU2AmIW2HCJ2PbQoKT5EaJMcx7wA7mdlnWeXFzWEsXgxrrAFz5kCnTumPcxzH+RZR\n6dQgP2iB/IKpQST1NLN34ufeQLtsY9Es2reH7baDV16BffZpsTjHcZwVnXzrYawCnAJsAkwipPX4\nphjhKVODHCZpECE4cBFQujSzu+wCL73kBsNxHKcE5FvT++9APfA8cADwrpmdXkHdsvUp3q32/vvh\n1lvhkebM1zuO47R+Sjkklc9gTIreUUhaGXjFzHqXotHm0CyDMXs29OoFc+fCSmliFB3Hcb5dVCrS\nuz7zodihqJphvfXg6KNhwYJqa+I4jtPqydfDWELwioLgGdUBWEiDl1TnVA1I/Qir6mXmMIZl7R9I\nw4p784FfmNmkHHI80ttxHKdIKjIkVRLh6VKD9AWmmNm8aFyGmlnfHLLcYDiO4xRJpZMPtoRlqUHM\nrB7IpAZZhpmNNbN5cXMsITrccRzHqTHKbTBypQbJZxB+DjxWVo0cx3GcZpEmcK8iSNobOB7Yvak6\nRaUGcVrGkiXwxRew5prV1sRxnCKoamqQFglPmRpE0jbAfUC/TNR3DlnNn8N44QX48EM4/PDmHb+i\nYQbHHRfiV558EnpXzZvacZwWUpE5DEnzJX0ZX/MT2/MlfZlS/rLUIJLaEVKDjMpqpzvBWBzTlLFo\nMZ9/DrfcUhbR30ouvhjefBOGD4cDDoBp06qtkeM4NUBZexiwzK12OA1utZcnU4NI+hNhvY13CS67\n9WaWKwV683sYc+dCz57w2WfQpk1zv8qKwcSJcMgh8OKLsO668NprsP32ft4cp5VScbfamHZ8UzMb\nIWktYDUzm1EKBdLSYrfazTeHe++FbQom2XW+/BI6pwqzcRynxqmoW62kCwmBdefGonbAXaVovKLs\numtIROgUxo2F4zg5SONW+2PgYGLUt5l9CKxWTqXKQiZzrdNyliyptgaO41SBNAbj6zgWZACSVi2m\nAUn9JE2VNE3S2Tn2by7pRUlfSTqzGNlFcdBBcNJJZRO/QtG/P9x9d7W1cBynwqQxGPdIuhlYXdKJ\nwNPAn9IIj6lBbiAswtQLGCBpi6xqnwL/A1yZWuvmsN56YVjKaWDJEjjllOK9oC6/HM48E26/vSxq\nOY5TmxQM3DOzqyTtB3wJbAZcYGZPpZS/LDUIgKRMapBluaTMbC4wV9KPilXeaSG//jVMnQo9ehR3\n3FZbwTPPwH77wVdfBaPjOM63nrSR3pMI2Wotfk5LrtQgy7nMOlXghhvgsceC+2z79sUfv8UWMGYM\n7LtvMBq/+lXJVXQcp7ZI4yX1c+AVQqzET4Cxkk4ot2JOGXn4Ybj00hDJ3bVr8+X07AnPPRfiXDyT\nsON860nTw/g1sL2ZfQogaU3gReC2FMfOArontr8by5qF55IqAXPnws9+BqNGwcYbt1xe9+5wySUt\nl+M4Tkmoai4pSS8CdWb2ddxuB4wxs4IzyJLaAG8R1sOYTeipDDCzKTnqXggsMLOrm5BVmvUwjjgi\neEwNHLjiRi9/9FGI4nYc51tPpdb0zri4bgdsDTxImMPoD0w0s+NSNVA4Ncg6wGuE2I6lwAJgSzNb\nkCWnNAbjkUdg6NCw3vegQXD88bDppi2X6ziOU4NUymBcmO9AM7uoFAqkpeQr7k2aFNxC77or9Dau\nuaZ0sld0Pv0UbrwRzjsPVir3kiuO4+Sj1SzRWkrKtkRrfT18/DF084X+Ssb8+WHYr3t3uO02WLlm\nll1xnBWOihoMSWsDvyEE3q2SKTezfUqhQFqqsqb3Qw+FZIXFximk4dln4YknYMIEGD8eOnaEfv1g\nyJDSTEZnuPbakBZl551LJzMNCxfCj38MXbrAX/8KbdtWtn3HcYDKr+n9V0Kg3UbARcBMwjoX335e\neAF22AG+//1w01u0qLjjFy8Oq9blYsYMaNcOTj4ZXn4ZHnggGKZSDuGMHBmG2rp3L1y31HTsCA8+\nGGI0Dj88LGDlOE7rxszyvoDX4/vERNmrhY5L1O1HMDjTgLObqHMd8DYwAdiuiTpWFRYtMvv73836\n9TPr2tXsl780W7rUzMxGjx7dUG/ePLPRo82uucZs0CCzbbYx69DB7KqrWq7D0qVmQ4aYPfCA2fz5\nBauPHj3a7LnnzNZe22zixJa33xIWLzY76SQbPXhw7v2PPmo2dKjZTTeF7zd2rNnMmWZffVV21Rr9\nfjWC65SeWtSrFnWK985U9+tCrzSDy/XxfbakA4EPgTXSGKNELql943GvSnrQzKYm6hwA9DSzTSXt\nDNwE9E0jvyKsskpwxT3iCJg1K2S8VejdjRkzpiEW5OGH4Q9/CIsN7bEHDB4MvXqF41vK11+HXFjX\nXw9HHw19+oSV8A44ILSRxZj77qPu3ntDr2jrrVvefkto1w5uvpkxQ4dSl2t/p06wdCm8/npw9/3o\no+DBNmRI7ujxp54KPbL6+obXN9+ERZ/22mv5+rfcEnpv2fXPPJMxU6YsH8vzxBMweXJI8d6lS8P7\nJpvAWmu1/HwUoNE1VSPUok5Qm3rVok6lJI3BuERSF2AIcD3QGUibB6JgLqm4/RcAM3tZUhdJ65jZ\nnJRtVI5u3eAnP8m9b+DA8CoH7duHG+iQIbBgQcjj9PjjIWBu5MjGdevrQ9mwYSHXU62zxx7hlZZF\ni8KrbVtYddXw3rZtuKnnYpddwu+WqZd5bbghTFkuHCgMI37wQVhEat68hvchQ3KvCX/JJSHFSnZM\nz1lnwcEHL1//yivD3Fg2TdW/6SYYPXp5/QcOhN12W77+U0+F5XXbtm08vLnnnrDllsvXf/bZ3Oeh\nqfqTJ8Mnn4Qhxw4dwnvHjiFjQCkejiqFWcMDxNKlQfdc82yLFoUHjGyaql9fH5w+0tYvVn5z6peQ\nNMkHH44f5wF7A0hKazDS5JLKrjMrltWewagFOnUKN5ZcNxcIF82AASGa+9tIvu+ei623Lq6XVaz8\nY46Burpw00nSVGxP//65HRCaqr/jjrD66o17SPX1TRvIzz6Dd94JdZJOIlttlbv+e+8Fx4tsmqr/\n5JNhbmrRouDYkHldcUU4F9lccAE8+miDgWnbNvTQBw/O/UAzfDg8/fTy5U3VHzYsxFZln5+LL4bD\nDlu+/oknwogRIVPzyisHfdq0CW7gufQ/88zgep9NU/WfeAKuuy59/WLlN6d+CWmWW62k98ys4Eyq\npMOAH5jZSXH7aKCPmQ1O1HkIuMzMXozbTwO/MbNxWbJah/+v4zhOjWEl8pJqroN82sbT5JKaBWxQ\noE7JvrDjOI7TPJrrw5n2af9VYBNJPWIOqqOAUVl1RgGDACT1Bb6oyfkLx3GcFZwmexiS5pPbMIiw\nNkZBzGyJpF8CT9KQS2pKMpeUmT0q6YeS/kNYN/z4or+F4ziOU3ZaTWoQx3Ecp7q0isxwkvpJmipp\nmqSzy9zWrZLmSJqYKOsq6UlJb0l6IroZZ/adK+ltSVMk7Z8o7y1pYtT52hbq9F1Jz0iaLGmSpMHV\n1ktSe0kvSxofdbqw2jol5K0kaZykUbWgk6SZkv4dz9UrNaJTF0n3xjYmS9q5BnTaLJ6jcfF9nqTB\nNaDXGZLeiPL+KqldDeh0evzfVfZ+UKoIwHK9CEbtP0APoC0hGnyLMra3OyGlezKyfRjBcwvgbODy\n+HlLYDxhaG/DqGem1/YysFP8/CjBW6y5Oq1LjIAHOhHWGNmiBvTqGN/bAGMJLtNV1SnKOAO4CxhV\nI7/fdKBrVlm1dbodOD5+XhnoUm2dsvRbiRDsu0E19QLWj79fu7j9d+DYKuvUC5gItCf8954EelZC\npxb/sOV+EaK+H0tsn0MTKUZK2GYPGhuMqcA68fO6wNRcugCPATvHOm8myo8C/lhC/R4Avl8regEd\nCWua7FRtnQhedk8BdTQYjGrrNANYM6usajoRgm/fyVFeE9dTlLU/8Hy19SIYjHeBroQb7qhq//cI\nS2X/KbF9PmFl1Cnl1qk1DEnlCv6rdC7y71j03DKzj4DvNKFbJuiwG0HPDCXTWdKGhB7QWMLFUTW9\n4tDPeOCWJc+IAAAGAUlEQVQj4Ckze7XaOgHXEP48ycm5autkwFOSXpX08xrQaSNgrqQRcfjnFkkd\nq6xTNkcCf4ufq6aXmX0IXA28F+XPM7Onq6kT8AawRxyC6gj8kNATK7tOrcFg1CJV8RSQ1An4B3C6\nhRUJs/WoqF5mttTMtic81feR1KuaOinkOptjZhPIHytU6d9vNzPrTfhjnyZpjxw6VFKnlYHewB+i\nXv8lPIVW9XrKIKktcDBwbxN6VPKaWp2QvqgHobexqqSfVlMnC7n4hhF60o8ShpuW5Kpa6rZbg8FI\nE/xXbuYoLCWLpHWBjxO65Qo6TBWMWAySViYYizvN7MFa0QvAzL4ExhAyE1dTp92AgyVNB0YC+0i6\nE/iomufJzGbH908Iw4l9qO55+gB438xei9v3EQxITVxPwAGELNlz43Y19fo+MN3MPjOzJcA/gV2r\nrBNmNsLMdjSzOuALwrxm2XVqDQYjTfBfqRGNn1BHAcfFz8cS1jfPlB8VvSY2AjYBXondwXmS+kgS\nITDxQVrGbYTxxuG1oJektTJeGJI6APsRxlCrppOZ/dbMupvZxoTr5BkzOwZ4qFo6SeoYe4ZIWpUw\nNj+J6p6nOcD7kjaLRfsCk6upUxYDCAY/QzX1eg/oK2mVKGtf4M0q65RZ2A5J3YEfE4bvyq9TKSao\nyv0iPLm+RVgz45wyt/U3gnfGYsLFcjxhwuvpqMOTwOqJ+ucSvA6mAPsnyncg3BjeBoa3UKfdCF3O\nCYTu57h4Ttaoll7A1lGPCQSPjfNiedV0ytJvLxomvat5njZK/G6TMtdvtc8TsC3hYWwCcD/BS6rq\nvx3BgeITYLVEWbXP1YVR/kTgDoK3ZrV1eo4wlzEeqKvUefLAPcdxHCcVrWFIynEcx6kB3GA4juM4\nqXCD4TiO46TCDYbjOI6TCjcYjuM4TircYDiO4zipcIPh1BySlkr6S2K7jaRP1JCu/CBJvykgYz1J\n98TPx0q6vkgdzk1RZ4SkQ4uRW0okjZbUu1rtOysebjCcWuS/wFaS2sft/UgkTzOzh8zsinwCzGy2\nmR2RLCpSh98WWb9VIalNtXVwWh9uMJxa5VHgwPi5UaqIZI8hPuUPl/SCpP9knvhjKplJCXnd4xP5\nW5IuSMj6Z8wiOymTSVbSZUCHmMn1zlg2SA2LIN2RkLtXdttJoh5vKmSEfUPS4xlDmOwhSFpT0ozE\n9/unwmI40yWdprCIzzhJLyokxMswKOo0UdJO8fiOCguBjZX0uqSDEnIflPQvQkSw4xSFGwynFjHg\nbmBAvLluQ1joJbtOhnXNbDfgIEIWz1x1diLk3NkWODwxlHO8me0U958uqauZnQssNLPeZnaMpC0J\nPY46C9l5T0/RdpJNgOvNbCtgHnBYnu+doRdwCCFR4aXAAguZZccScv5k6BB1Oo2QbwzgPOBfZtYX\n2Ae4Kub7AtgeONTM9m5CB8dpEjcYTk1iZm8QVgcbADxC/nTlD8RjptCwBkA2T5nZF2b2FSF30u6x\n/FeSJhBuxN8FNo3lyfb2Ae41s89jO18U2fYMM8v0dl6P36sQo81soYWMrV8AD8fySVnHj4ztPw+s\nJqkzIcHhOQprlYwB2tGQ8fkpM5uXon3HWY6Vq62A4+RhFHAlYfW8tfLUW5z43JRhWW79Akl7EYzB\nzma2WNJoYJUidUzTdrLOkkQb39Dw0JbdbvIYS2wvpfH/Nte6DAIOM7O3kzsk9SXMDzlOs/AehlOL\nZG68twEXmdnkZhybzX6SVo9DM4cALxAytH4ejcUWhOWAM3ydmBh+hjCMtQaApK5Ftt1U+Uxgx/j5\n8CbqFOLIqNPuhNXg5gNPAIOXNS5t10zZjtMINxhOLWIAZjbLzG5IUzfPdoZXCENREwjDS+OAx4G2\nkiYD/x94KVH/FmCSpDvN7M24/9k4zHN1kW03VX4V8AtJrxNSUzdFPrlfSRoH3AicEMsvJnyviZLe\nAH6XR7bjpMbTmzuO4zip8B6G4ziOkwo3GI7jOE4q3GA4juM4qXCD4TiO46TCDYbjOI6TCjcYjuM4\nTircYDiO4zip+D+zNhM8a9e+zAAAAABJRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x1ae972678d0>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Compute the moving average loss to smooth out the noise in SGD\n",
|
|
"plotdata[\"avgloss\"] = moving_average(plotdata[\"loss\"])\n",
|
|
"plotdata[\"avgerror\"] = moving_average(plotdata[\"error\"])\n",
|
|
"\n",
|
|
"# Plot the training loss and the training error\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"plt.figure(1)\n",
|
|
"plt.subplot(211)\n",
|
|
"plt.plot(plotdata[\"batchsize\"], plotdata[\"avgloss\"], 'b--')\n",
|
|
"plt.xlabel('Minibatch number')\n",
|
|
"plt.ylabel('Loss')\n",
|
|
"plt.title('Minibatch run vs. Training loss')\n",
|
|
"\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"plt.subplot(212)\n",
|
|
"plt.plot(plotdata[\"batchsize\"], plotdata[\"avgerror\"], 'r--')\n",
|
|
"plt.xlabel('Minibatch number')\n",
|
|
"plt.ylabel('Label Prediction Error')\n",
|
|
"plt.title('Minibatch run vs. Label Prediction Error')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Evaluation / Testing \n",
|
|
"\n",
|
|
"Now that we have trained the network, let us evaluate the trained network on the test data. This is done using `trainer.test_minibatch`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Average test error: 1.74%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Read the training data\n",
|
|
"reader_test = create_reader(test_file, False, input_dim, num_output_classes)\n",
|
|
"\n",
|
|
"test_input_map = {\n",
|
|
" label : reader_test.streams.labels,\n",
|
|
" input : reader_test.streams.features,\n",
|
|
"}\n",
|
|
"\n",
|
|
"# Test data for trained model\n",
|
|
"test_minibatch_size = 512\n",
|
|
"num_samples = 10000\n",
|
|
"num_minibatches_to_test = num_samples // test_minibatch_size\n",
|
|
"test_result = 0.0\n",
|
|
"\n",
|
|
"for i in range(num_minibatches_to_test):\n",
|
|
" \n",
|
|
" # We are loading test data in batches specified by test_minibatch_size\n",
|
|
" # Each data point in the minibatch is a MNIST digit image of 784 dimensions \n",
|
|
" # with one pixel per dimension that we will encode / decode with the \n",
|
|
" # trained model.\n",
|
|
" data = reader_test.next_minibatch(test_minibatch_size,\n",
|
|
" input_map = test_input_map)\n",
|
|
"\n",
|
|
" eval_error = trainer.test_minibatch(data)\n",
|
|
" test_result = test_result + eval_error\n",
|
|
"\n",
|
|
"# Average of evaluation errors of all test minibatches\n",
|
|
"print(\"Average test error: {0:.2f}%\".format(test_result*100 / num_minibatches_to_test))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Note, this error is very comparable to our training error indicating that our model has good \"out of sample\" error a.k.a. generalization error. This implies that our model can very effectively deal with previously unseen observations (during the training process). This is key to avoid the phenomenon of overfitting.\n",
|
|
"\n",
|
|
"**Huge** reduction in error compared to multi-class LR (from CNTK 103B)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We have so far been dealing with aggregate measures of error. Let us now get the probabilities associated with individual data points. For each observation, the `eval` function returns the probability distribution across all the classes. The classifier is trained to recognize digits, hence has 10 classes. First let us route the network output through a `softmax` function. This maps the aggregated activations across the network to probabilities across the 10 classes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"out = C.softmax(z)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let us a small minibatch sample from the test data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Read the data for evaluation\n",
|
|
"reader_eval = create_reader(test_file, False, input_dim, num_output_classes)\n",
|
|
"\n",
|
|
"eval_minibatch_size = 25\n",
|
|
"eval_input_map = {input: reader_eval.streams.features} \n",
|
|
"\n",
|
|
"data = reader_test.next_minibatch(eval_minibatch_size, input_map = test_input_map)\n",
|
|
"\n",
|
|
"img_label = data[label].asarray()\n",
|
|
"img_data = data[input].asarray()\n",
|
|
"predicted_label_prob = [out.eval(img_data[i]) for i in range(len(img_data))]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Find the index with the maximum value for both predicted as well as the ground truth\n",
|
|
"pred = [np.argmax(predicted_label_prob[i]) for i in range(len(predicted_label_prob))]\n",
|
|
"gtlabel = [np.argmax(img_label[i]) for i in range(len(img_label))]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Label : [4, 5, 6, 7, 8, 9, 7, 4, 6, 1, 4, 0, 9, 9, 3, 7, 8, 4, 7, 5, 8, 5, 3, 2, 2]\n",
|
|
"Predicted: [4, 6, 6, 7, 8, 9, 7, 4, 6, 1, 4, 0, 9, 9, 3, 7, 8, 0, 7, 5, 8, 5, 3, 2, 2]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(\"Label :\", gtlabel[:25])\n",
|
|
"print(\"Predicted:\", pred)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let us visualize some of the results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Image Label: 9\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztXdly4loSLAwYDBivfSdi/v/vpr2wY8BmHm6kOlWqI7AN\nMrQyI05I0DYWNKmqU0tWY7vdmiAI9cLFT1+AIAjVQ8QXhBpCxBeEGkLEF4QaQsQXhBpCxBeEGkLE\nF4QaolXB31ChgCD8HBrRk7L4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI\n+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiC\nUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBD\niPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4\nglBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEO0fvoChGqx3W6P8rOffU0++uca\njUb28zjn5/a5xug1dv1+dG1l18qLf+7i4sIajUa2/GP/nqLnjg0RX8iwDym/ezPAa358fOQWnjOz\nkDA496/jz/n3o9fhn4+Iy9fir43X+/t7eGw0GtZqtZKr2Wxm1xStqiDi1xyeyBEpIpJ8FSAJE4bX\ndru1ZrOZIwM/bjQaGRlT1+bJxETDe2RC+6O/Jr82m42t12vbbDaF1Wg0rNPpZOvy8jJ33m63rdls\nWrPZzG4EzWbTzPKW/9gQ8WuMlJubIgQs8nfw8fERkgbPmVlGBk8OWEt/TXyd/Pt802CCeavtLXnq\n2nC+Wq2yI6/1em0XFxd2dXVlvV4vPHY6ncz6f3x8WKv1LwVxc6oKIn5NsYv0qfVdvL+/ZyTxJFqv\n12Zmpa4yiF92fbhZ8E0DRMM1RJ4Hbkq8/LWuVitbLpf29vaWHfm82WzaYDAorPV6nf2dy8tLu7y8\nLHgoVULErzlS+/rUfva7rj6Iv1qtMtLgfLVa2Xa7tXa7nVyNRiNnqffdY8O6NhoN22w2Bdcd5+v1\nunBNfFwul7ZYLMI1n8+t1WrZzc1Ntt7e3jLSNxqNgncCS99sNo8STE1BxK8x9iW934N/B5vNpmA1\neZlZZhHb7XZ2zsSO9t241kajYe1221qtVnazYKLhGrC8a88Ex+LHi8XCZrOZzWYzm8/n2TlWu922\nu7s7m8/ntlwuc6THjceTnr2RqiDi1xBlbj6vKKj1XeLDZX57ewut5na7zQXHmNjb7TYjPltpPm80\nGtlNg2MADL935+NqtUpadFj1yWRi0+k0PLbbbZvP5wVL32w27fLyMhfIA+lxQ5XFF5IoyztHz5VZ\ndU90EKXMIh6C+BGZcG5mhWg4L++q8zUy8b3HgHMzS5J+s9kkb0ie+BHpJ5NJ9rfa7bZ1Oh3rdrt2\ndXVlV1dX1u/37fLyMnPtW61Wdt2y+MJOlBF31zGVj45I76PZm83m21/QzWZTcO/Zld5utxlZo5Vy\n9bFgRVMxAlxDmasfXRcew71fLBaZK8+fC/9fsBeB1+12u9n18e9GnskxIeKfGXZF3X3AKyo08e6x\nt5wc1faPP0P86Iv8/v6eC5b5czPbGdz7SgEN9v24htRWIRXcwzkH9/Acfh/v16cF8Zq4gbTbbVut\nVrmtTJWkNxPxzxK+0MQH4SJrFuWno0KUKMXG5+/v79+69mOn87Cf9qk8HHENqZsjrH7q+rxHAOKz\nxeb/H45p4PewjZHFFz4FT3wmt7fS0Tl/sVP5anyp/drX4qe+xOwCRzei7XZbKNph8vL7j4p4OD3m\ni3eQK9/lLZUVGPnCncjV55sI/zzI3+l0shuG9xaqgoh/ZogCcZFL7r+cEYkjgqfccBy/Y/FT2QJe\nZlYo0+XH/DpRjCOqg/9Mye6uct1UyS6TNyI/SN/tdnP/LxxjkcUXShGVl3rLwuT1JOaKM195tuv5\nfYhf9gWOrLT/4vvmGt/dxqmvqIegrDPO/05ZqbJfuDFEsYVdFp9dfaT6cPOQxRd2gr+gZe5kKjLN\nC5Hp1IpeC/X0n73m6HHqGLWt8jEiiSe+P/fNL1EK1F9L2XnZzcf//0TE56Ag1ylUBRH/xOC/QP45\nuI1sjX3N+HdXZPnx5f0K8fm9mX2/53wXOauA76HHQvZgV2uu34ZU1ZUHiPgnhl3puqiyjFNMUeNI\nisTRY07ZNZvNLPftG132fS+78NkvPFvTKDBXBfmjPnoQ+OrqKmvMub6+tuFwaMPh0G5ubnLHwWBg\nV1dX1u12s1bdKskv4p8g2E30Ry4i8cfZbGbL5XJnAC+VquM9p5llxTAXFxdZ5VvVe9Gyz8anKXFT\nAI5xrT5r4FOHqNAD8a+vr7OGndvbWxsOh3Z9fW39ft96vZ51Oh0RX/hj0VIpucViYdPptLBQOopq\nshSxU2k0LCbOodpFUwT8yhcd6TZe+BuI6vOW4tDkZ+KzW4/zfr9v/X4/Iz2s/O3tbUZ8/Az680V8\nISz+YKs9n89tPB7bZDKx8XhcOF8sFskmFCZ3qohlu80r4Pi0WtV7UQ+u/PPpOcQfjk1+eEKtVqvQ\nE9Dr9UJX//b21u7u7uz6+jqr3RfxhQxRnTcH3WDdX19fbTQaZQuPF4tFcpvA5aG+3h/nIBOIztas\n3W5XLhjhsdlsCjl9fF4XFxfZ+zgW+b3FRzMOFqw5E59d/cFgUGg8QteeiF9zMPG9+MN0OrXxeGyj\n0cheXl7s5eXFXl9fs3MQP1Wjv4sA2NOb/Qnucaccqud+CpC3YlEL3Aw4yu7Jfygw8WHx0YXX7Xat\n1+uFxIfF7/f72Q2Dm4lQjlwVRPwTA1ePscWHAIQn/vPzsz0/P9vT05M9Pz/bfD4PC2O4yCRSr2Uy\nIZIPi9/pdDLdONS7/xRWq1WO9NxYwwU+xyR/ivggfZnF7/f7ST1AEf8vRqqYBfBFH4jYI4jHLj72\n9bzPR2trKteN7jX+0nFUutPpWK/XyxYIj4WbwrE/H3/E+dvbW3ZzxOfkifNdsnt5bp+n588E0Xmc\ng+SI3g8Gg+xmgM+zrCqxKoj4P4CyAhRYL1h4DtzB0r++vtp4PLbpdFpQewG5U4MboAQTiVXAeiG/\njAAUnx/b4kels3yEYg28IbjJ3w08+s8oavLBjRFEjhYCeTc3NzYYDKzX6xVy9V7nn/9+VRDxfwgc\nWOMve0R8kB3WHjcDCEJExE81qbTb7YzI2Jf6x3Bdo2MVxOfiHB+raDabuS3Qd4nvyccR+2gv3u12\nMyuO6D2fDwaDnMX3ufpoqs5PZEpE/B+AJztbtRTxsZ8fjUY5cUcmPgpvoiKTyGLxnpRdUbb+fGTN\nuGMhJRbC0lrYBnlCfRZRHT+IH71/bIOY5HDno4XPtNvtZpJbp0B6MxG/ckSkZ4tWRvynpycbjUYF\niWfU0HOjB+/fudAEJaWwSLyGw6FdXV0VVG9YR+7YxPcdh/7czLLPB9f1FYufagDC5xZte7rdbq44\nhxduAnwDjXL1uEFFtf5VQsT/AfgOu88QfzweF/rkUZmHqH2UawZJer1eRnJUk2FPent7m0XuU+vY\neXzev0eVh9vt1haLRZZe9C70ZxCRn2sXQHgO4vFNE8U5OGfX3o/Q8kU6qa7BqiDi/xDY4nuRh13E\nT0loQWwSFoRTTt5Vvbm5sfv7+8Lq9XrhFqGqlBNX5kW9Bh8fHzabzTL3GRb/M55Iqm2XiZ9y7UF2\n33yDBY8p0vs7haAeIOL/AHy02iu7pIj/+/dvm0wmSREL1KqbFV19zjWjceTu7s4eHx/t8fHRfv36\nZb9+/bJer5cMDFbhkuLGl+ow3Gw22b75Oxbfky8iPiw+3HvOyfPi57vdbvLzO4WSZ0DErxiRiAYv\naMzP5/PcQhfefD7PXodfE2AXn/epcFkjy8Vf4n6/X5rH/u4Xd5cIx8XFRfbZcEEOKvS4K4/7C1Kf\nh4cnol9l1XdMdHbzeZ/f6XSyv1N2/GmI+BUD6SqWZOKFdB2IDjcXX3Ig5S4y6WGteHGHGAJPl5eX\nSVf0GEgJZW63W1utVrmbnL/pjUajLLsxmUzCOgZ8znzkzyrVWYfiHN87z0eQvKy77tRIHkHErxhM\n/GhqCxPfp+pg2cq+YJFr73vDOfqM4hIfGT9WysnHNnx34HK5zFUq+jUej+3l5cVGo1FWwLRcLgtK\nt3xkoHLRp+mwer1ewarz4jQd3zj9duMU9vFlEPErRkR8zsuniO975aN0EFfmcX09iM+uKivAeIvv\nXx+PD/0ZRNr/HNtItR7zc7ssvj9n4nPUHtshdu+jyH2/389VMsLic5PNqZPeTMSvHCniQ0gDX2g/\npinVWeeLQuDGwuJjr8r7VG/xU8Tn4yHfP1t5LxDCvQlR2zE+J/yMJz7+Bv89/3nh5shqOV41J6px\nQC+9T9elAoynSnozEb9yMPG56w4WLbL42ON7y+Wj0bBmID5bfK4hx5ccOecy4vvzQ7z/SHMAR/95\ncMvxy8uLTSaTgs5gVMCEv+XhLT5H7Nmd94U5WNga+XJen+o8ZdKbifiVo8ziw7qVufp+j++j1N7i\nM/Fh8dm1hcWv0mKVSYOnSpWfnp7s6enJptNpOBjEu/opMPE5VceFTKmS3Ovr60yTwK+qAqOHgohf\nMb5K/H0sPled4YvNe3xE9Lk5JxXcO+b7j4aBIGcPV58t/vPzs/3+/dt+//5ts9ksOfByHwVgdvVh\n8eEN3d/fZz3zqfp7TOw9RqqzSoj4FYMDW5G77/f30TRWbsBhrfZWq1VovIk6yXwpKbuqh8jT+1Zj\nfhypCvGKAni85vN5chKPWXkvfaPRyKU4vTYeAp8+BQoPCd7R3wARv2JEYprQ1EPxDh77oYzQxOMy\nXL+Gw6Hd3d1l1h1k56AUk/3QbqrvQfDnEBdJrfF4XMjTc4DTtzMDKPTx4iL+OBgM7P7+PvuMEOz0\n3XS7AnfnDhG/YkSBLbj8+PJHs9fZorGr6rvHQHxO24H4XN/ONe6HktHG+2M33i8IhnIKkxcLieLn\ncBOMiO/JzzMAIsGRwWBgd3d3uc8IWQ58Tr6wqaptUJUQ8X8A3uJ74vvBivyFZ4sfVeZ5i4+OsejL\nzMQ/5B7V39h81D5VnINUnXftfUrTE94s31YLncBIZAT1DBzMww2SXXrOz3+13/+UIeJXDF937l39\n+Xyei1azFn6K+L7gZJcLm4pIH+r9RcMiUZLMsYxowcpzqa539fF3PNjiI53p9+ogPu/p/efkPQW5\n+sK3Ee3xvcX3E3S8q+/z0CzhzF/mlKt/zEEZfGNjhWDELlCkhJoFH8yDa+8n+kb9ChHY4vtyZb5B\n+sf4nDhP772ivwkifsXYh/jRIAyu0/eVZ0jV3d/f5yw9R/I5uOeLfvj8EO/PW3y8N5+29FV5o9Eo\n83h4m4DHUUrTn3uL74uXcDOMFryiKCh4bum6XRDxfwC7XP3UpFyWx+aSXBD/4eEhq8zjtB7vXdvt\ndkiYY1h8X6uA/Tvy837xQBDOz/sbYKrIyJcs4zPgPD0KmPwWAOcpj0gWX/gWODjFX2yuWY9kpWHp\noqKdsj27l/nah+D+Z/zjKKqO87e3t0KQjgN30UwA5Osx9DN10+M6Bn7vfPQluBzIQ8CT5wXgyAE9\n9oB+Svf+2BDxfxBl5bHIS+MIpCr/8MU1s3BCLuvQ73NN/suOc75hRcM3ka7DilpseT/vqxMjojOi\ndB0/7vf7WVYDY6s42Il9POfqfcrulCSyjgUR/4fhv1ye6B6e+NgiwFpBzMITH8/tQ/yo+YevzU/h\n5SOX3aYW3xAi4pfl6Tm+wcpC3FbrtfB8TUNZd90pSWAfEyL+CSCyLpG1Nysq+EBxFl9eED8i/Xq9\n3pv4Kd04VN9FQpi4nrLKPJYWw9GrDKWsvdm/Fh/E5+65qKEmOmJuAHsJpzbsogqI+BWj7IvEVpVJ\nz+ccNYfFn81mWeT54+MjJD7Od03C8QEttoTNZtM2m00u3eYX192D3PyYc/p8nsrTl1n8wWBQkAhn\nrYHoiGlA3FLLrv6p6N4fGyL+D2DXl6jM3Y8sPkiPf/Oa9F8hfqr1FGIZKcvOvfJ8xHlqG+KJ798z\nXx+08TibgYXuQ1/K7FuQuckpCooeOttxahDxfwhlHWRR8wng9/hMei6ciaz9Pnt83/nnz9frdSFg\nx49BdLboID5b9ihll1IZYsDiozgHxP/Pf/5j//zzjw2Hw3D0FY5s2VPy139zUA8Q8U8UqUAfCM4z\n4VOKNl7d5jPET03SWa1WWS7eH8fjcRasw2IlYZQg++BdKpAXHXmPj8EgDw8P9s8//9h///tfGw6H\nhUGX7NJj8EbKlf9bie4h4v8AYGWigRdvb285C4g0GYOFLFarVSHN5gd08A1gH1ff9/jzWq/XoQAm\nztmdjxaGfkTZA/z9ssyC76FnZSHk6cs8lr+tEOerEPErRlRdxiq4ZlaaLmOLxKW/3mJ5pRuQf9eo\nqVQ/O87X63XBzfetxDzskiP1/P598U30XHQcDof2+PiYtdXyvDq+Vp+eE/IQ8SsGE9+PaALxfZoM\nVorLdqNqPHaZvVeAvfYu4uP6osAeovqpoB6781FBDl6fJcL8UAvfMuzPr6+vM+J7mXAvLuILcoQ/\nEPErBqekuHX0+vo6G4+1XC6t3W7bcrnMkd6nu3gb4GWouBfg8vLSlstlVodehrJU3sXFRaZ9H6Xx\nOB+fGnPFgqCRgpAf0e3XYDCwh4eHzOKzWjBrB0YBO5H/D0T8itFoFCWwQfzlcmlmlvsCm/1p6rm4\nuMjlt6ObgBexjCa27rq+KNqN51AnEI3qhsy1jzWkLD5ai1k0AxF4nvDLjxHJj4jvhUXOTfm2Soj4\nFcPv8UF8iE3gxuBJv16vrdlsFsiOI77ccMfx837tU0MQleoy8X3cIdIOiBpsuK3Yj6L2zTKpI7rt\neMJNNMMueg/CH4j4FYNdXSY+SGP2J0Dn21ubzWZSfspb1LI89S54svgagyjrELn1/hxgV5/jHNxC\nzIU3/BjBUFbKZVefif83V959FyJ+xYgsflS1tt1uc6T301pSBINV/U6OelcBS1ke3pfa+mPk6qOc\nFvX07AFEy98QcFPAtib1XoQ/EPErRkR8r5vvCc0y1a1Wq2BtvYQ1w1tbPMd/h5879Hvlwhszyyw9\n3HevGwjiw/r75WfXcWut8vT7Q8SvGJHF85NwzfL5dC7wWSwWSelqPz8uInY0iIIfH/I9RjECnukH\nwvNAC6Tn2OVnq86ttF4eXNZ9f4j4PwAObjHpfSotGoCJyjhfjstBtjI3PNqTc0rwu5Y/ygZwOpBH\nU/OEWk98P+aLo/6RGKZI/zmI+BWDXX3e18ML4Co5donh6nK+nGvisdjq++0Ct/TyMrODWXu8R9/h\nh/dWRnzMrQPBo8WKOVEfvbAfRPyK4V19sz+k900lTPrZbGaDwSDZ7ooja/ZFC2277Bp/fHxkjw9l\n8VONPvtY/NR4MBbN8PX32tt/DiJ+xeA8tlkxvYcvN5N+sVhkpI/64Dudji0WC2u321kjTGp23Xq9\nzhGF03OHsphRExLWPsQvq9xDjYOvKpS1/xxE/IoBUrTb7ZylBznh0oL0vred9eym02ku4NVsNrOS\n2VSe3RcHcVXgod5fRHzc0LBX98RHh91gMEi2BPN+PqrOE/n3h4hfMXx3mo+8c1ONb2mFqi5kqznY\nhddcLpehHj3WarUyM8t5A4ceGOHFPLgufx+LH1UccvMNPsfoKOwHEb9i7CIYB/hAGJap8tJRPojW\n6XQKk3j4HOIdZsVy4EO9P1wTByexeLpPlKu/urpKNghpL384iPgnCCaPV53tdDqFqTKcBVgul2F+\nH6k+3/EHD+NQefCo+5ALcqLptN5ridx3ufKHhYh/guBcOLYDeN635/rgIHTtogYaeAxmedKzdt8h\nrj0SGuG9PGvc++0K7+FVmHM8iPgnBh8cY9Kz4IaZ5fbQ2DtHKra82u12YZqt7wP47vX7a+KOutvb\n20JNvu+lV9Du+BDxTxBM/larlbOAXPvO+X4/WTZ1jrZdDhYy4Q5x7b4XIZpYC/37yNVXW+3xIeKf\nGPhLz49RZIPFpOd9vB8v7bMDjUYjN7eei2KOscfnmnwQ34+l9q4+v2+R/jgQ8U8QbO0gV8UluK1W\nKxxYybLbkbT129ubmVlu2CYr1xza4sPV53l2t7e3uWBf5OrjdaKjcBiI+CcIFr0wy7fMpnrfsRC5\n5/p9frzdbnMTdtniHwI+2BhZfN915+fX4TMQjgcR/8SwjwhGGTabTSEQ6It1fF6crequ1/cuuE+5\nYV/PQhoo1MHiJhxusVWevjqI+GeOaLwW6+jD4vPwSh5n5cdT7wJc+VQBEarwELlHoQ5uBkx2n8IT\nqoOIf6ZISVtFQzQ8+UF8aOB/hfi++QYrIj728qjeQ0Win1Ir8lcHEf+MESntlBGfp9n6iTefJT72\n8H6xAi5H730EP+qnF6qDiH+GSAX2UmOz2NrzjHoeT71vHz53F/J+HhF8ztN7i4+f8/30h24SEnZD\nxD9jlElrsULvMVx9LwmGvTwTH0U62OPD4nPjjYQ0fgYi/pkiRXo/My/l6n81uGdmOeJ3u92caCb6\n6r3FZ1ffl+VKOqt6iPhniCivz9p60R4/iup7i78P2OJjTj3LY3tXn2vyYfXxOtFRqAYi/pnBu/Re\nT8+79V6xB2OtfYAvsvgRKVlYwyvpREE9RPI5ii/8PPS/cGZgjbyo7342m9l4PC5d0+k0R34/0CNq\nkMHiQRi+QIfde5YEO2QTkHAYiPhnCNTke2391Wpls9ksk+aaTCYF0k8mk0Jaj7X9fSeg18f3U3Ai\n4nPenqfciPinAxH/zMDVeRy4Q/R+Op0mLf1oNLLpdJqba48gX2TxuSoP50x6T3yQv6zPXjgNiPhn\nBuzvWUiDA3fT6TRn7b3Vn81mhVZdL+XlVXI57+5VgDmVB4vvx13hNYTTgYh/ZogsPgfxIrLD2oP4\nfr69D+55a4+AHoiPgF3K1eebg/b4pwkR/8yQsviI2jPp/Q1gNBrZYrFIau6bpXXxsVgiO3L1MQmH\ntfRF/NODiH9mgMVHcI9FNTioF1n98Xhsy+WydKimJ77Xxmdrjhw+W/zr6+twfJaCe6cFEf/EkKqe\n8004ID0q8tjiT6fTXM7eF+6UgWW9PNE7nU6uMIer8rjtNhqEIdKfFkT8E0Wq7Rak9xV5sPggvK/O\n+2yBDu/h+Xhzc2N3d3dZWS5y9hzEi2bWi/inBRH/BFHWfZdqt4XFR3EOxmlz8A6IymVxDqFM1OD7\n/bsnPtfg++m1kUKPcBoQ8U8UqT041+EjsOeJP5vNMuJHlXlmsYSWWZ74vV4vq8HHQhMOW3zo5jHx\nfSGQSH9aEPFPEN7aoxb/sxZ/uVwma/FT2nns6oP4UMe9u7vLtdymXH3p4p8+RPwThSc9N+Ls2uPD\nE0jt8b2LzwT1Fh/u/f39vd3f39vNzU2uaAd7f5+vly7+aUPEPzGkeuyjXvuUxWcdfXb1GZ6YIKzf\n48Pi393d2ePjow2Hw5x+Hlfp+bSd9venCxH/ROF77MsktTzx/bw8tvgpFx/k964+B/RAfMy6R7oP\n5740V2Q/XYj4FWMflRsmup91DyuPdB3OOVfvW3W9wo4v0uFinahIh137fr+fq+TjAh1JaJ0PRPwT\nA+/jo+ULc7yMlid7FNTjclxfXecbbFj7vmxuvaz7eUHEPzFwLT6q83i/jqg95+p5Lw/SY/lxW2bl\n2vjYu6em3ESkF84PIv6JIdVvjxVZfB6Djf28t/i+155183ih7NYTH0E7Feb8HRDxTwgcyWeLz/p5\nXKDjXX0/HIOzAQAH8bx+HuSykZf3xN+VqtMN4Hwg4p8IuGgnsvgpwUw/HGOz2RTSgSmLD+LDwrMS\nLtRzyojPEOnPCyL+CcDLZXO/vRfa8K4+q+mA+HidaMSWmeWID8vOPfZePWeXxRfpzw8i/g/Dp/fK\n6vGjdlvv6u8ah5Wy+FFBThTVj8gvnB9E/B+Ct8g4+l57VteBqg7v81kfHwG9VKqt0WjkVHI5P89i\nmei1j6ryWHFX5D9fiPg/AL/35vOI8JPJxEajkb2+vtpoNLLJZJLrwGOV3EgSm1e3282p5fBYa36M\n51gjP9VvL/KfH0T8iuGj7f6x39ezjt7r62s2EAPufiSP7WWz+Pzq6io39YbHXnGrLaw+LL+Xyo7I\nL5wPRPwfQDQGCwU3XkATXXej0ShbLKedsvjI03ORTqvVyunjoa/+9vbWbm9vM+Kz4g4W7/cjd99M\nkf1zgoj/A/CNN1yPz65+yuL7IRqr1SokPkftsfyAS/TZY0EeG+49nyPCz4U8XmJLOA+I+BXDW3s/\nA8933fk9/ng8Dmv4PfG9YCbIyxZ/OBxmxH94eLD7+/usCQcWnmW1YfGjdl4R/7wg4v8AUq22iOaz\nZLa3+JPJJFeLz7X52+02NwyDq/JYDht7fFh8iGw8PDxYv98vxAZ4qV7/74CI/wNg/Ty2+Kk+ewzD\neH19tel0WsgEeF187+pzVR5cfS+pdX9/b4+Pj9br9cKafFXs/V0Q8Q+MqCCH4ctx/WJd/JQ2fhnY\n1WfSI2fPuvisj886+cLfDxH/CCiTx+ZqPL/m87mNRiN7fn7OJtsiZYca/F1Ayy0q8kB6n7Pn/Dzv\n3YV6QMQ/AlJNMphr73XyuAHHE3+xWBRm2JchIj5r593c3OQq80B8RebrBRH/wGBL74UyuUAHEXse\nbono/cvLS8Hir9frL1t8uPIo1OEZ9j5aL9QDIv4R4KP2XhobFp+j9SjHxThrDL5Ekc5XXH0Ia/jc\nfWTxRfx6QcQ/Arw0NqffELWHxYeFf3l5yVx8bAO4LPere3wvk826+ClZbOHvh4h/BKSKdLzFh6v/\n+vpqT09P9vv372yUNS8Q/6t7fJ6Iw7r43uIL9YGIf2BEgzCY/NEe/+XlxZ6enux///tfrjIP7bas\nrLMLZXtJkxiPAAAGzUlEQVR8uPosoa09fj0h4n8Su8jnNfNYHQcqucjVYx+Phc47VsuNdPG5uMaf\ns/59tHq9XliGq6h+vSDiHxjI1Xt1XOTux+OxPT8/Z+W3SNl5FR2fDWDSsxY+D7VotVo2HA7t4eGh\nMMPet9ZGI62F+kDEPzBg7bnengN1nK4bj8ehoEaZLn6j8e9EW7jqvpNuOBza4+NjgfhQzvXEj+rv\nhb8fIv6BAGIy8X1bLXfZweIz8VkeO7L2Zv9afJ5th+Adlp9sm8rZi/T1hoh/ADAxUZ23Wq1ssVhk\nKTvO1Y/H42x5i48gnic+N+HA4mOopZfNuru7ywlrRDn7SEVHqA9E/G8iasrxFh/u/dPTk41Go9xk\nW1+dx4E8v8z+7PFB/Ovr65yCDlJ2XkqLXf1Ii09Wv14Q8b+BiPR+j+9TdijQQccd6+P7tF1q7h1S\ndUx89NOjMs9337HFT7XdCvWBiH8glO3x0Xjz+/dvG41GuRHXfOSBGP51AVh87PFB/IeHB/v165fd\n3t4W9v08FgvEN7PcUcSvF0R8h33y9L7zjl1zWHHuvuMA33g8znL7vLxSboqUfvoN99nDzY+GY/Ao\nLIYIX0+I+J8AyB0V2OCIElzo32MPH8242zXDPtK2Q7Udp/O8oCYKdLwGvodIX1+I+DvgPQCea8eV\neXjMarhI1/k5dyyu6avydg3E8OW2eOxJj1x9Kngn0tcbIn4J/DBLszzxfSPNcrnMtdb6ybYshc0i\nmWUTbf1i0peRv6xAR6QXRPwEItL7SbbosuPlR13B1S+rzvMW3wtm4og0HhM95epHZbl4fUEQ8XfA\nD7Zk4rMENhbr37Orzxbfd+7xPp9dfVhsdt8j0keuvt/jK3IvMET8EkTTbDld55V0WAabJ9v6PX5Z\ngY5ZPMqah2OkSO9n2rO7L9ILDBE/QMrNj1x9FsmM5LMii+9vKHweufqcwttnj+9r8VWWK3iI+AlE\nFhlDMHjwhR9zldrbc66+DKyJz1NwsPz8el+cw/t6tvYK7AkMET9AWZEOBDZSwy254265XGaEj2Sz\nIjJiT8/FOXy8ubmxh4cHu729teFwmKvDj3rstbcXIoj4CaTm2MPNj0ZdpaL4SN35fXxUnddsNjNL\nz+Ou/NgrJj5abjlvH5FfNwABEPEdvLX3EfjI4vNwy8lkklPf4WEYfh/viY+WW5bF5uGW6L7zE3HQ\ngMNqueq6E8og4geIyB8JZnqLj5Ser8OPLD7grXJk8Xmi7XA4LMy7g8VPufkiv+Ah4ieQ0sVnBdzU\nHp+n37KqTmTxfT1+GfF//fqVyWPz8nt8kV3YBRE/gA/qeYsfufq8x0dRDlfm+Wh+RHqO6LOrz223\nw+EwzOWzq4/Xj46CYCbiF7DLzfcjriOL74tyoiIdMyuQfl+LX6ayq9JcYR+I+CXwOXx/I2C3HzeD\n1WqV/b7vhovKcbnQptlsZuIZ0MGHig6i+oPBoPA7u9pvBcFDxD8CUnt4nnLD1pqPiOAjiNfr9ZJ5\nekXtha9CxD8CWDjDk5Q77KLuupubG7u7uwuJH5XhqlBH+ApE/AMjtW9nIQ1fhssLwTyIZnqLL2sv\nHAIi/hHg03PcV89aedFADATzvMXnOnyp5ArfhYh/BOwS0uA6fF+Mg0o9VOZFe/xUua8g7AsR/8CI\novccwGOLH03B4WPK1Y9IL/ILn4GIfwRwBJ+tPQZhQPqaZbG5Dp+9gMji898RhK9AxC9BqqQWajhM\n4MFgYDc3NznpKy+F1W63rd/v54gOt56Jj30/ynHhLShPLxwKIr5D5KpzxR3KaRGEg5RWo/HvaKvl\nclmoquMjfjdag8Egq7/3MlpS0BEOCRE/AJN/u93m3OvLy8ss+s6kx1irt7e3nN4dl9MiuIeqvGiB\n9Jh+w9NtBeFQEPEDMPH9c2zx39/fc6Tv9/u2Wq1y0Xy/8PupBdceBT1crSeLLxwKIr5DRHp+Hhaf\nLT2s+HA4tM1mUyja4bp6BPhSi6W0ZfGFY0HET4DJj3O06l5dXeVIPxgMbLFY2GKxsPf391xlnT9n\nQkeLtwi+AUcWXzgURPwAPk+O7jyuloPl7vf7uS49eAIpFZzUaKyUJLbksYVjoLFrLPQBcPQ/cEj4\nz8Nr30ciHbwA35LL5/su/nl/Lgh7IvzCiPiC8HcjJL4iRoJQQ4j4glBDiPiCUENUEdVXNEoQTgyy\n+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiC\nUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBDiPiCUEOI+IJQQ4j4glBD/B98ADVKMyzD/gAAAABJ\nRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x1ae97291b00>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Plot a random image\n",
|
|
"sample_number = 5\n",
|
|
"plt.imshow(img_data[sample_number].reshape(28,28), cmap=\"gray_r\")\n",
|
|
"plt.axis('off')\n",
|
|
"\n",
|
|
"img_gt, img_pred = gtlabel[sample_number], pred[sample_number]\n",
|
|
"print(\"Image Label: \", img_pred)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"source": [
|
|
"**Exploration Suggestion**\n",
|
|
"- Try exploring how the classifier behaves with different parameters, e.g. changing the `minibatch_size` parameter from 25 to say 64 or 128. What happens to the error rate? How does the error compare to the logistic regression classifier?\n",
|
|
"- Try increasing the number of sweeps\n",
|
|
"- Can you change the network to reduce the training error rate? When do you see *overfitting* happening?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"source": [
|
|
"#### Code link\n",
|
|
"\n",
|
|
"If you want to try running the tutorial from Python command prompt please run the [SimpleMNIST.py](https://github.com/Microsoft/CNTK/tree/v2.0/Examples/Image/Classification/MLP/Python) example."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"anaconda-cloud": {},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.5.2"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 1
|
|
}
|