CNTK/Tutorials/CNTK_201B_CIFAR-10_ImageHandsOn.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNTK 201B: Hands On Labs Image Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This hands-on lab shows how to implement image recognition task using [convolution network][] with CNTK v2 Python API. You will start with a basic feedforward CNN architecture in order to classify Cifar dataset, then you will keep adding advanced feature to your network. Finally, you will implement a VGG net and residual net similar to the one that won ImageNet competition but smaller in size.\n",
    "\n",
    "[convolution network]:https://en.wikipedia.org/wiki/Convolutional_neural_network\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this hands-on, you will practice the following:\n",
    "\n",
    "* Understanding subset of CNTK python API needed for image classification task.\n",
    "* Write a custom convolution network to classify Cifar dataset.\n",
    "* Modifying the network structure by adding:\n",
    " * [Dropout][] layer.\n",
    " * Batchnormalization layer.\n",
    "* Implement a [VGG][] style network.\n",
    "* Introduction to Residual Nets (RESNET).\n",
    "* Implement and train [RESNET network][].\n",
    "\n",
    "[RESNET network]:https://github.com/Microsoft/CNTK/wiki/Hands-On-Labs-Image-Recognition\n",
    "[VGG]:http://www.robots.ox.ac.uk/~vgg/research/very_deep/\n",
    "[Dropout]:https://en.wikipedia.org/wiki/Dropout_(neural_networks)\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "CNTK 201A hands-on lab, in which you will download and prepare Cifar dataset is a prerequisites for this lab. This tutorial depends on CNTK v2, so before starting this lab you will need to install CNTK v2. Furthermore, all the tutorials in this lab are done in python, therefore, you will need a basic knowledge of Python.\n",
    "\n",
    "CNTK 102 lab is recommended but not a prerequisites for this tutorials. However, a basic understanding of Deep Learning is needed.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "You will use Cifar 10 dataset, from https://www.cs.toronto.edu/~kriz/cifar.html, during this tutorials. The dataset  contains 50000 training images and 10000 test images, all images are 32x32x3. Each image is classified as one of 10 classes as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/201/cifar-10.png\" width=\"500\" height=\"500\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 1\n",
    "Image(url=\"https://cntk.ai/jup/201/cifar-10.png\", width=500, height=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above image is from: https://www.cs.toronto.edu/~kriz/cifar.html\n",
    "\n",
    "## Convolution Neural Network (CNN)\n",
    "\n",
    "Convolution Neural Network (CNN) is a feedforward network comprise of a bunch of layers in such a way that the output of one layer is fed to the next layer (There are more complex architecture that skip layers, we will discuss one of those at the end of this lab). Usually, CNN start with alternating between convolution layer and pooling layer (downsample), then end up with fully connected layer for the classification part.\n",
    "\n",
    "### Convolution layer\n",
    "\n",
    "Convolution layer consist of multiple 2D convolution kernels applied on the input image or the previous layer, each convolution kernel output a feature map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/201/Conv2D.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 2\n",
    "Image(url=\"https://cntk.ai/jup/201/Conv2D.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The stack of feature maps output are the input to the next layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/201/Conv2DFeatures.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 3\n",
    "Image(url=\"https://cntk.ai/jup/201/Conv2DFeatures.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Gradient-Based Learning Applied to Document Recognition, Proceedings of the IEEE, 86(11):2278-2324, November 1998\n",
    "> Y. LeCun, L. Bottou, Y. Bengio and P. Haffner\n",
    "\n",
    "#### In CNTK:\n",
    "\n",
    "Here the [convolution][] layer in Python:\n",
    "\n",
    "```python\n",
    "def Convolution(filter_shape,        # e.g. (3,3)\n",
    "                num_filters,         # e.g. 64\n",
    "                activation,          # relu or None...etc.\n",
    "                init,                # Random initialization\n",
    "                pad,                 # True or False\n",
    "                strides)             # strides e.g. (1,1)\n",
    "```\n",
    "\n",
    "[convolution]:https://www.cntk.ai/pythondocs/layerref.html#convolution\n",
    "\n",
    "### Pooling layer\n",
    "\n",
    "In most CNN vision architecture, each convolution layer is succeeded by a pooling layer, so they keep alternating until the fully connected layer. \n",
    "\n",
    "The purpose of the pooling layer is as follow:\n",
    "\n",
    "* Reduce the dimensionality of the previous layer, which speed up the network.\n",
    "* Provide a limited translation invariant.\n",
    "\n",
    "Here an example of max pooling with a stride of 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/201/MaxPooling.png\" width=\"400\" height=\"400\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 4\n",
    "Image(url=\"https://cntk.ai/jup/201/MaxPooling.png\", width=400, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In CNTK:\n",
    "\n",
    "Here the [pooling][] layer in Python:\n",
    "\n",
    "```python\n",
    "\n",
    "# Max pooling\n",
    "def MaxPooling(filter_shape,  # e.g. (3,3)\n",
    "               strides,       # (2,2)\n",
    "               pad)           # True or False\n",
    "\n",
    "# Average pooling\n",
    "def AveragePooling(filter_shape,  # e.g. (3,3)\n",
    "                   strides,       # (2,2)\n",
    "                   pad)           # True or False\n",
    "```\n",
    "\n",
    "[pooling]:https://www.cntk.ai/pythondocs/layerref.html#maxpooling-averagepooling\n",
    "\n",
    "### Dropout layer\n",
    "\n",
    "Dropout layer takes a probability value as an input, the value is called the dropout rate. Let's say the dropu rate is 0.5, what this layer does it pick at random 50% of the nodes from the previous layer and drop them out of the nework. This behavior help regularize the network.\n",
    "\n",
    "> Dropout: A Simple Way to Prevent Neural Networks from Overfitting\n",
    "> Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov\n",
    "\n",
    "\n",
    "#### In CNTK:\n",
    "\n",
    "Dropout layer in Python:\n",
    "\n",
    "```python\n",
    "\n",
    "# Dropout\n",
    "def Dropout(prob)    # dropout rate e.g. 0.5\n",
    "```\n",
    "\n",
    "### Batch normalization (BN)\n",
    "\n",
    "Batch normalization is a way to make the input to each layer has zero mean and unit variance. BN help the network converge faster and keep the input of each layer around zero. BN has two learnable parameters called gamma and beta, the purpose of those parameters is for the network to decide for itself if the normalized input is what is best or the raw input.\n",
    "\n",
    "> Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\n",
    "> Sergey Ioffe, Christian Szegedy\n",
    "\n",
    "#### In CNTK:\n",
    "\n",
    "[Batch normalization][] layer in Python:\n",
    "\n",
    "```python\n",
    "\n",
    "# Batch normalization\n",
    "def BatchNormalization(map_rank)  # For image map_rank=1\n",
    "```\n",
    "\n",
    "[Batch normalization]:https://www.cntk.ai/pythondocs/layerref.html#batchnormalization-layernormalization-stabilizer\n",
    "\n",
    "## Microsoft Cognitive Network Toolkit (CNTK)\n",
    "\n",
    "CNTK is a highly flexible computation graphs, each node take inputs as tensors and produce tensors as the result of the computation. Each node is exposed in Python API, which give you the flexibility of creating any custom graphs, you can also define your own node in Python or C++ using CPU, GPU or both.\n",
    "\n",
    "For Deep learning, you can use the low level API directly or you can use CNTK layered API. We will start with the low level API, then switch to the layered API in this lab.\n",
    "\n",
    "So let's first import the needed modules for this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "\n",
    "from cntk.layers import default_options, Convolution, MaxPooling, AveragePooling, Dropout, BatchNormalization, Dense, Sequential, For\n",
    "from cntk.io import MinibatchSource, ImageDeserializer, StreamDef, StreamDefs\n",
    "from cntk.initializer import glorot_uniform, he_normal\n",
    "from cntk import Trainer\n",
    "from cntk.learner import momentum_sgd, learning_rate_schedule, UnitType, momentum_as_time_constant_schedule\n",
    "from cntk.ops import cross_entropy_with_softmax, classification_error, relu, input_variable, softmax, element_times\n",
    "from cntk.utils import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/201/CNN.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 5\n",
    "Image(url=\"https://cntk.ai/jup/201/CNN.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we imported the needed modules, let's implement our first CNN, as shown in Figure 5 above.\n",
    "\n",
    "Let's implement the above network using CNTK layer API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_basic_model(input, out_dims):\n",
    "    \n",
    "    net = Convolution((5,5), 32, init=glorot_uniform(), activation=relu, pad=True)(input)\n",
    "    net = MaxPooling((3,3), strides=(2,2))(net)\n",
    "\n",
    "    net = Convolution((5,5), 32, init=glorot_uniform(), activation=relu, pad=True)(net)\n",
    "    net = MaxPooling((3,3), strides=(2,2))(net)\n",
    "\n",
    "    net = Convolution((5,5), 64, init=glorot_uniform(), activation=relu, pad=True)(net)\n",
    "    net = MaxPooling((3,3), strides=(2,2))(net)\n",
    "    \n",
    "    net = Dense(64, init=glorot_uniform())(net)\n",
    "    net = Dense(out_dims, init=glorot_uniform(), activation=None)(net)\n",
    "    \n",
    "    return net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the above model we need two things:\n",
    "* Read the training images and their corresponding labels.\n",
    "* Define a cost function, compute the cost for each mini-batch and update the model weights according to the cost value.\n",
    "\n",
    "To read the data in CNTK, we will use CNTK readers which handle data augmentation and can fetch data in parallel.\n",
    "\n",
    "Example of a map text file:\n",
    "\n",
    "    S:\\data\\CIFAR-10\\train\\00001.png\t9\n",
    "    S:\\data\\CIFAR-10\\train\\00002.png\t9\n",
    "    S:\\data\\CIFAR-10\\train\\00003.png\t4\n",
    "    S:\\data\\CIFAR-10\\train\\00004.png\t1\n",
    "    S:\\data\\CIFAR-10\\train\\00005.png\t1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# model dimensions\n",
    "image_height = 32\n",
    "image_width  = 32\n",
    "num_channels = 3\n",
    "num_classes  = 10\n",
    "\n",
    "#\n",
    "# Define the reader for both training and evaluation action.\n",
    "#\n",
    "def create_reader(map_file, mean_file, train):\n",
    "    if not os.path.exists(map_file) or not os.path.exists(mean_file):\n",
    "        raise RuntimeError(\"This tutorials depends 201A tutorials, please run 201A first.\")\n",
    "\n",
    "    # transformation pipeline for the features has jitter/crop only when training\n",
    "    transforms = []\n",
    "    if train:\n",
    "        transforms += [\n",
    "            ImageDeserializer.crop(crop_type='randomside', side_ratio=0.8) # train uses data augmentation (translation only)\n",
    "        ]\n",
    "    transforms += [\n",
    "        ImageDeserializer.scale(width=image_width, height=image_height, channels=num_channels, interpolations='linear'),\n",
    "        ImageDeserializer.mean(mean_file)\n",
    "    ]\n",
    "    # deserializer\n",
    "    return MinibatchSource(ImageDeserializer(map_file, StreamDefs(\n",
    "        features = StreamDef(field='image', transforms=transforms), # first column in map file is referred to as 'image'\n",
    "        labels   = StreamDef(field='label', shape=num_classes)      # and second as 'label'\n",
    "    )))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us write the the training and validation loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#\n",
    "# Train and evaluate the network.\n",
    "#\n",
    "def train_and_evaluate(reader_train, reader_test, max_epochs, model_func):\n",
    "    # Input variables denoting the features and label data\n",
    "    input_var = input_variable((num_channels, image_height, image_width))\n",
    "    label_var = input_variable((num_classes))\n",
    "\n",
    "    # Normalize the input\n",
    "    feature_scale = 1.0 / 256.0\n",
    "    input_var_norm = element_times(feature_scale, input_var)\n",
    "    \n",
    "    # apply model to input\n",
    "    z = model_func(input_var_norm, out_dims=10)\n",
    "\n",
    "    #\n",
    "    # Training action\n",
    "    #\n",
    "\n",
    "    # loss and metric\n",
    "    ce = cross_entropy_with_softmax(z, label_var)\n",
    "    pe = classification_error(z, label_var)\n",
    "\n",
    "    # training config\n",
    "    epoch_size     = 50000\n",
    "    minibatch_size = 64\n",
    "\n",
    "    # Set training parameters\n",
    "    lr_per_minibatch       = learning_rate_schedule([0.01]*10 + [0.003]*10 + [0.001], UnitType.minibatch, epoch_size)\n",
    "    momentum_time_constant = momentum_as_time_constant_schedule(-minibatch_size/np.log(0.9))\n",
    "    l2_reg_weight          = 0.001\n",
    "    \n",
    "    # trainer object\n",
    "    learner     = momentum_sgd(z.parameters, \n",
    "                               lr = lr_per_minibatch, momentum = momentum_time_constant, \n",
    "                               l2_regularization_weight=l2_reg_weight)\n",
    "    trainer     = Trainer(z, (ce, pe), [learner])\n",
    "\n",
    "    # define mapping from reader streams to network inputs\n",
    "    input_map = {\n",
    "        input_var: reader_train.streams.features,\n",
    "        label_var: reader_train.streams.labels\n",
    "    }\n",
    "\n",
    "    log_number_of_parameters(z) ; print()\n",
    "    progress_printer = ProgressPrinter(tag='Training')\n",
    "\n",
    "    # perform model training\n",
    "    batch_index = 0\n",
    "    plot_data = {'batchindex':[], 'loss':[], 'error':[]}\n",
    "    for epoch in range(max_epochs):       # loop over epochs\n",
    "        sample_count = 0\n",
    "        while sample_count < epoch_size:  # loop over minibatches in the epoch\n",
    "            data = reader_train.next_minibatch(min(minibatch_size, epoch_size - sample_count), input_map=input_map) # fetch minibatch.\n",
    "            trainer.train_minibatch(data)                                   # update model with it\n",
    "\n",
    "            sample_count += data[label_var].num_samples                     # count samples processed so far\n",
    "            \n",
    "            # For visualization...            \n",
    "            plot_data['batchindex'].append(batch_index)\n",
    "            plot_data['loss'].append(trainer.previous_minibatch_loss_average)\n",
    "            plot_data['error'].append(trainer.previous_minibatch_evaluation_average)\n",
    "            \n",
    "            progress_printer.update_with_trainer(trainer, with_metric=True) # log progress\n",
    "            batch_index += 1\n",
    "        progress_printer.epoch_summary(with_metric=True)\n",
    "        \n",
    "    #\n",
    "    # Evaluation action\n",
    "    #\n",
    "    epoch_size     = 10000\n",
    "    minibatch_size = 16\n",
    "\n",
    "    # process minibatches and evaluate the model\n",
    "    metric_numer    = 0\n",
    "    metric_denom    = 0\n",
    "    sample_count    = 0\n",
    "    minibatch_index = 0\n",
    "\n",
    "    while sample_count < epoch_size:\n",
    "        current_minibatch = min(minibatch_size, epoch_size - sample_count)\n",
    "\n",
    "        # Fetch next test min batch.\n",
    "        data = reader_test.next_minibatch(current_minibatch, input_map=input_map)\n",
    "\n",
    "        # minibatch data to be trained with\n",
    "        metric_numer += trainer.test_minibatch(data) * current_minibatch\n",
    "        metric_denom += current_minibatch\n",
    "\n",
    "        # Keep track of the number of samples processed so far.\n",
    "        sample_count += data[label_var].num_samples\n",
    "        minibatch_index += 1\n",
    "\n",
    "    print(\"\")\n",
    "    print(\"Final Results: Minibatch[1-{}]: errs = {:0.1f}% * {}\".format(minibatch_index+1, (metric_numer*100.0)/metric_denom, metric_denom))\n",
    "    print(\"\")\n",
    "    \n",
    "    # Visualize training result:\n",
    "    window_width            = 32\n",
    "    loss_cumsum             = np.cumsum(np.insert(plot_data['loss'], 0, 0)) \n",
    "    error_cumsum            = np.cumsum(np.insert(plot_data['error'], 0, 0)) \n",
    "\n",
    "    # Moving average.\n",
    "    plot_data['batchindex'] = np.insert(plot_data['batchindex'], 0, 0)[window_width:]\n",
    "    plot_data['avg_loss']   = (loss_cumsum[window_width:] - loss_cumsum[:-window_width]) / window_width\n",
    "    plot_data['avg_error']  = (error_cumsum[window_width:] - error_cumsum[:-window_width]) / window_width\n",
    "    \n",
    "    plt.figure(1)\n",
    "    plt.subplot(211)\n",
    "    plt.plot(plot_data[\"batchindex\"], plot_data[\"avg_loss\"], 'b--')\n",
    "    plt.xlabel('Minibatch number')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.title('Minibatch run vs. Training loss ')\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "    plt.subplot(212)\n",
    "    plt.plot(plot_data[\"batchindex\"], plot_data[\"avg_error\"], 'r--')\n",
    "    plt.xlabel('Minibatch number')\n",
    "    plt.ylabel('Label Prediction Error')\n",
    "    plt.title('Minibatch run vs. Label Prediction Error ')\n",
    "    plt.show()\n",
    "    \n",
    "    return softmax(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 116906 parameters in 10 parameter tensors.\n",
      "\n",
      "Finished Epoch[1 of 300]: [Training] loss = 2.062444 * 50000, metric = 75.3% * 50000 13.316s (3754.8 samples per second);\n",
      "Finished Epoch[2 of 300]: [Training] loss = 1.675133 * 50000, metric = 61.7% * 50000 13.772s (3630.5 samples per second);\n",
      "Finished Epoch[3 of 300]: [Training] loss = 1.520789 * 50000, metric = 55.4% * 50000 13.674s (3656.7 samples per second);\n",
      "Finished Epoch[4 of 300]: [Training] loss = 1.421881 * 50000, metric = 51.4% * 50000 13.668s (3658.2 samples per second);\n",
      "Finished Epoch[5 of 300]: [Training] loss = 1.338381 * 50000, metric = 48.0% * 50000 13.675s (3656.3 samples per second);\n",
      "\n",
      "Final Results: Minibatch[1-626]: errs = 43.3% * 10000\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYVOX5//H3B1QQRGwR1Ii9FyJYEhULxl6TGHTtGvWb\noF8T9KeJ8auoSTR2jYolKPaNvWLvvS0YNSAaESwgqCBIFdj798d9xjk7O7s7u+zszM7er+uaa04/\nzzNnYO59qsyMEEIIIYRy0anUCQghhBBCSIvgJIQQQghlJYKTEEIIIZSVCE5CCCGEUFYiOAkhhBBC\nWYngJIQQQghlJYKTEEIIIZSVCE5CCCGEUFYiOAkhhBBCWYngJFQESbWSzmrhuRMk3ZhaPzK5Xr/W\nS2HLSVojSc/JpU5LaJik1yU92sJz/yVpbGunqcB7tzjdIRRLBCehbKSCglpJ2zZwzGfJ/odydlny\naonaPOe26rwOkqok/b41rxkaJmlE6rvU2OvGpq9WsMX5Dhr+PSyFmMMklJ0lSp2AEPKYCxwCvJre\nKGlHYDVgXp5zlgYWtvB+G1D8H4ZDgE2AK4p8n+CuBZ5Kra8FnAtcD7yU2v5xK95zAC3/oT8MUCum\nJYR2LYKTUI4eBX4t6SQzSwcNhwBvAyvlnmBm37f0Zma2oKXntleSupnZnFKno1jM7A3gjcy6pP7A\nX4DXzOyOQq4hqauZ5QuEG7pnS4NjzGxRS88NoRJFtU4oNwZUAysCu2Y2SloSOBC4gzx/Yea2OZF0\ndrJtHUk3SZou6VtJN0rqmnPuhAaK97tLuk7S15JmSLpZ0nI55+4n6RFJX0iaJ+m/kv5PUqfUMc8B\newOZtiO1ksan9ndJ0jtO0lxJkyTdK2mtPPk8LrnHPElvStqyqQ80VV22g6RhkqYAnyX7bpL0SZ5z\nzpZUm7OtVtI/JO0v6b0kDe9L2r2J+68saYGkM/PsWz+57uBkfQlJQyV9mHwWX0t6SdIuTeVzcUj6\nUtJdkvaWVCNpHnBEsu84Sc9KmpKk6T1Jx+S5Rp22G5J2T/K2X/J5fiFpjqQnJK2Rc26dNieSNsh8\nLsnr4+Ter0rqm+feh0gamxzzTpKPFrdjkdQr+W5MTa45WlJVnuOOkDRK0nfJv69/S/pdav9Skv4q\n6aPkOl9JekHSDi1JV+g4ouQklKMJwOtAFfBEsm0vYFngX0AhbTcyxet3AeOBPwH9gGOBKcDpeY5N\nE3AVMB0Yilf9DAb6ADunjjsK+A64BJgFDMSrD3oAf0yO+SvQE6+S+kNy7VkASRAzMrlmNXB5cu6u\nwKZAOnA4FFgGr7Kw5Pr3Slq7wL+8hwFTgXOAbqm858t/Q9sHAL9MrvUdcBJwj6Q+ZjY9303NbKqk\nF4BBeOlF2sF4ddxdyfo5+LO6HngLf+Zb4s/umaaz2GIGbA7cjOftWuA/yb7BSVrux6v/DgCGSzIz\nG5FzjXyGAvOBv+NB92nATdT9HjX0ef8G6ApcDXTGn/k9ktY3MwOQ9EvgNrxU8Y94yeKtwKRG0tQg\nSd2Bl/Hv6z+Az4GDgNslLWNm/0yO2zfJx2PAdfgfu5sAPwOuSS53Pv7v9RpgNP7vYGvgJ8CLzU1b\n6EDMLF7xKosXcCSwCP8hGgx8C3RJ9t0JPJ0sfwI8lHNuLXBWan1osu36nOPuBabmbPsEuDEnHbV4\ntUDn1Pb/l6Rvn9S2LnnycQ3+w71katvDwPg8xx6d3OukRj6XNZJjpgLLprbvm6RnrwI+11rgeUA5\n+0Y0kK6hwKI8n/FcYM3Uts2S7YObSMNxSVo3ztn+PvBUan107rNtpe9W/ySdRzSwf3KSvu3z7Mv3\njJ8F3svZ9hrwaGp99+Seo3K+R6cm91o7ta0aGJNaz7SD+gLontr+6+Tcgalt44CP0unEg9va9DUb\n+Wxy0/3H5B4HpLYtgQc/3wBdU9/zL5u49ljgrtZ+nvGq/FdU64RydRf+1/0+kpYB9gFub+Y1DP+L\nLu0lYMXkmk253uqWSFxDEgz8cAOz+ZllSctIWhH/q7MbsGEB9/gl8BVeStOUf5nZzNT6S3gpzNoF\nnGvAP81scXtmPGVmE364qNl7wMwC0nAf/tkdlNkgaRNgY7w0LONbYBNJ6y5mOltirJm9nLsx5xn3\nlLQS/lf/RpKWKuC6w3O+R5kGuYU8t9vNbHbOuT8886Tqbz1gRDqdZvYUHrC0xJ7ARDN7IHW9hcCV\nwHJApifdt0BPSQMbuda3wOb5qihDaEwEJ6EsmdnXwNN4I9hf4t/Ve1pwqU9z1jNVD8s3lQTgvzlp\nmo3/hb1mZpukjSXdL+lb/Ef6K7xIHbwIuynrAOOsbsPfhnyWk55vk8Wm8pIxocDjCk5DYnpTaTCz\nb/BqmUGpzQcDC/Dqkoyz8B/ADyW9K+lCSZstXpILVq/tDXgvMUnPSZqN53Vqkk7h1U5Nyf3Mpifn\nFvLc8p1L6txM25V8vY7+m2dbIdYAPsyzfSye7sw9rwQmAk9Jmijpn5J+nnPOGUAv4OOkLcz5kjZu\nYbpCBxLBSShnd+ClFL8FHjOz71pwjYbaYix2t01JPfG/oDcD/g8v3fk52bYmrf3va3HzMjfPtoZK\nUjoXIQ3/AtaXtHmy/mvgGTOb9kNizF7CA7ajgffwNhej8jVALYJ6n4+kDYEnge5424m98GecKekq\n5BkvzmdWtO/v4jKzSfh3/xd4u6mfA09KuiZ1zLP48/wNHtz8D/COpEPbPsWhPYngJJSzTAPEbfBA\npS0JLy7PbvCGgquQLYHYCf8L9kgzu8rMHk3+M/6W+hoKAj4GNpDUUDBQbNPxkopcaxbhXg/gJSUH\nJT1O1sfbWtRhZt+a2c1mdiiwOvAucHYR0lOI/fH2FnuZ2XAzezx5xuXS/Xxi8p6vGqylVWMT8WeT\nayP8e5y5J2a2wMweMrPBeFXTTcDxklZNHTPNzEaYWRXeoHwc3qYphAZFcBLKVlKN8lv8h+nhEiTh\neEnpHm2D8RKFTHfRRXgQk+42vFRyXK7Z5K/muRf4EXBiayS4BT7G2w1smtkgaRW8R0qrMrMZeO+r\nQXiVznzgwfQxklbIOWcOXj3RJXXMsklX20KqVBZXpuQi/YxXxAdNK0RRR181s0/wtiVHKdVFXt69\ne70GT2zco3i39/1T11sC/45+C7ySbMt9VoY3cIbkeeU5Zhbee64LITQiuhKHclOnuNrMbm3owDaw\nFPCMpLvwxq2/A14ys0eS/a/iJQ+3SPpHsu0w8v8g1QCDJF2Cd0udlVznFnw8jUslbYM3eFwG2AW4\n2sxaKyhrqBrgX8AFwANJHrrjAeE4vNdUa7sT7/Y6GHgip4EvwBhJz+Of1zRgK3x8m3+kjvkF3svo\nKPzzK6bHgfOAxyQNx0uZjsd70dQbDDCPtqh+OQP/XF+WdAuwMv5d/Q8t+wP0arzL/R2SrsLbvRyM\nfx9+m2p4e5ukLngvsC/wkpMTgTeSoAm8rcljeI+l6Xg3432AC1uQrtCBRHASyk0hf2nmGxNicec1\nyXe9E/GxRc4BlsR7C/0wxoqZTZO0Nz7GyV/w/3xvxbuZPpFzvWFAX/wH9Q940fgjZlYraU/8BybT\n+PcbPEh5r4D8FZrvvMckeTgAuBQPUj7BxxlZn/rByeKmAeAhvG1Hd+r20sm4AtgP7wrbBf+c/gxc\nnOeezdXYOXnzYGbvS/o1/nwvwX+EL8NLfYYVcI+G7tnQ51jouT/sM7N7JB0OnIk/w3F4kDwYWDXv\nFRq5t5nNljQAH5flaHzcnbHAoWaWfmY34W1JBuNB22Q8WDwndcxl+ACEu+PP8xO8K/XlBaYrdFBa\n/J6FIYQQyk0yOuyHZrZ/kweHUGZK3uZE0unyYbhnyoeHvl9SvsZYDZ2/nXxo7FHFTGcIIZQj+ZD/\nnXK27YEP5PZcaVIVwuIpecmJfC6Kanz0wSXw4Y43BTYys3xdH9Pn9sTrpj8CeplZMerIQwihbEna\nAO8JVY1XrWyCd9n9Eti
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14663f064e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzsnXeYFGXSwH8FEgQERQRMeKgYUERBUURMfIqYM2JOeCoq\noicGVDBnjKCiB+p5cCbMKIr5MJL1wISgIAIiWTJb3x/V7fTMzuzOzM7uzs7W73n66ek3db3Tszs1\n9VbVK6qK4ziO4zhOvlCjsgVwHMdxHMeJ4sqJ4ziO4zh5hSsnjuM4juPkFa6cOI7jOI6TV7hy4jiO\n4zhOXuHKieM4juM4eYUrJ47jOI7j5BWunDiO4ziOk1e4cuI4juM4Tl7hyomTESJSJCI3Ztl3pogM\njVyfFYzXLncSZo+IbBPIc0Vly1KdEZEPRWRKjseM++zlM0n+Tg4IPpf75/AeWf8dO05F4MpJNSSi\nFBSJyL4p2swK6l9LqNLgyIaiJH1zun+CiPQQkd65HNMpneCz8lCOhiuPPTXSGjPyd1EkIutF5FcR\nGS0iB5SDTKlIJmvG74mIdBOR/iXco8L3LhGR/gnvceL73bSiZXLykw0qWwCnUlkJnAp8Gi0M/hFv\nCaxK0mdDYF2W99sRU1DKk1OBXYAHy/k+TuHyDvAMIEBL4GLgfRE5XFVHV7QwqvqRiGyoqmsy7Ho4\nJvtNSerK8ndcVhS4EPgzSd3iCpbFyVNcOanejAJOEpHLVDWqNJwKjAOaJHbI4h9ktO/abPtWVUSk\nnqquqGw5nIz4XlWHhxci8gowBbgcSKqciIgAtVV1dXkIlOXfneR4vFzykqouzKSDiNQB1miS3Wpz\n8Xfmf6v5hS/rVF8UGAFsChwSFopILeBEYDhJ/rklrlWLyICgbDsReUpEFonIYhEZKiJ1E/qmWvev\nLyKPi8gCEVkiIk+LyMYJfY8WkTcCM/sqEflRRK4XkRqRNh8ARwCh70iRiPwUqa8TyPudiKwUkTki\n8pKItEwyz57BPVaJyJcismdpb2hkuWx/ERksIvOAWUHdUyIyI0mfASJSlFBWJCIPicgxIvJ1IMM3\nItK1lPs3FZG1InJDkrodgnEvDq43CEzs3wfvxQIR+UREupQ2z2xJ5xkmtG8nImNFZIWI/CQif0/S\npraI3CQiPwRj/iIid4lI7VzJrarfAAswK0p43/AZnSoi32BWxq5BnYjI5cEzWykic0XkscTPdND2\nerEl1D9F5D0RaZ2kTVKfExHZW0RGichCEVkuIpNF5NKgbhhmNYkuVa1PkP/GhPH2EJG3gr/BZSIy\nRkT2TmgTfsb3FZGBIjI/uPdIEdk04zc3BZE5dxeRW0VkNmZp2UhEzk71d5bFPJKO4VQ+bjmp3swE\nPgd6EPtFeDjQEPgPkI7vRvgr5nngJ+AaoB1wPjAPuDZJ2ygCPAIsAvpjSz8XAy2AgyLtzgaWAfcB\ny4GDgZuBjYCrgza3Ao2wJanLg7GXAwRfgG8GY44AHgj6HgLsCkQVh9OABsBjgcxXAy+JyLaqup7S\nGQzMx8zp9SJzT+VLkKy8M3B8MNYy4DLgRRFpoaqLkt1UVeeLyEfAycAtCdWnYGb854Prm7BnNQT4\nCnvme2LP7r3Sp5gVZ1P6MwxpjD2v5zFF+WTgURFZrapPwV/WiteBfYHHgW+BNkAfoBX2/pUZEdkE\n2AT4IaGqSyDXI5jyMjMoHwKcCQzFlhdbApcCu4tIp/AzJCK3AP2AN4C3sPf+HaBWEjHiPiMicgg2\n9znYZ3kusDNwJPAw9n5sAfwf9nlOaUUJxmsNfAwsAe7EPit/Bz4Ukf1V9auELg8DC4EBwN+w9/wR\n7H9JOmwaPL8o61R1SULZDcBq4B6gDrCG2HsR/TurH8xjlwznUWwMJ09QVT+q2QGcBazH/hlejK3z\n1gnqngPGBK9nAK8l9C0Cboxc9w/KhiS0ewmYn1A2AxiaIEcR8AVQM1L+j0C+IyNldZLM41Hsy65W\npOx14Kckbc8J7nVZCe/LNkGb+UDDSPlRgTyHp/G+FgEfApJQNyyFXP2B9Une45XA3yJlbYLyi0uR\noWcga+uE8m+AdyPXExOfbRk/U0XAQ6W0SfcZfhDMoXekrBYwAfgt/KwApwNrgY4JY14Q9N8n1Wev\nlHkMwSyKTYAOwJgk8hQF994xof9+QV33hPJDgvJTgusmmLXl1YR2twbton8nBwT33z+4roH9EJgO\nbFTCXB5O/GwlyB/9O345+MxtEylrjn3Jf5DkM/52wnj3YYpDSnkin/eiFMfUhDkXYQph7Qz+zjKd\nR7Ex/MiPw5d1nOexX/dHikgD7JfXvzMcQ7FfalE+wX4dNUij/xCNt0g8SqAM/HWDyFq+iDQITMj/\nDWTfKY17HA/8jv26K43/qOrSyPUn2C/PbdPoq8ATGvwHLAPvqurMvwZV/RpYmoYMI7H3rntYEPya\nbI1Zw0IWA7uIyPZllDNtMnyG6zAlIey7FvuMNQXaB8UnAtOA70Vk0/DAlBsh3vKWCedhn5X5mGWx\nI3CfqiY6WX+oqt8llJ2IvbfvJcg0EbMWhTIdgilcDyf0fyAN+fbArBUPqOqy9KaUmsCqeAjwsqr+\nHJar6lzMarVfwt+xEnk2AZ8ANTEFvzQUOA6z6kSPc5K0fUqT+8cU+zvLch65+Ft1ygFf1qnmqOoC\nERmDOcHWx36VvZjFUL8kXIdLD5sQLK2kEgH4MUGmP0XkN+wfMPCX2fk27J97w4T+jdKQbzvgO413\n/E1F3Nqzqi4OLNCbpNEXYub9spBs/XtRaTKo6h8i8h623BCGkZ6C/cp/OdL0RuAV7Iv9G+Bt4F+B\nElQuZPgM56jqyoSy7zGl42/Al9jSzU6YIpGIYopMNryKKbGKWXX+l0QWSP6cWwEbY4pNSTK1CM6J\nn/0FIpJ02S7CdsFY/yulXbpshimI3yepm4b9T9g6eB2S+PmM/r2nwyeankPszAzqsplHSeM7lYgr\nJw7Yr4ongM2Bt7L8NZbKF6PEte50EJFG2DryYuB6zKS9CvsFfSe5d+wu61ySfZGl+nVWsxxk+A8w\nVER2U9UpwEnAe9EvA1X9RES2A44BDsWsBX1E5O+qmvNkZeX0DGsAX2P+Dsnel2wdHGer6vtptEv2\nnGtgvlanppApmSJVFSm3v/cEkr3H6dTlYnynEnHlxAH7Rf04sDeR5YAKQrBfmx/9VSBSH1OU3gyK\nDsR+kR2jqmMj7bZLMl4qJWA60EFEamp6Tq25ZhH2izqRv5XDvV7Bnmf3wOlwB8xiEYeqLgaeBp4W\nkXqYaX4A5siZaw4k/WcIsIVYbo/ol8eO2PMNnZenA7up6gflIG+2TMccZT/VksOKw2WHVkR+vYtI\nE0q3PkzH/m52BUpSotJdrvgdWIG9v4nsjPlmVIVIlkKZh4OHEjvYMgqWFGkA5lBa0VwgIlFF+WLM\nojAquF6P/TOOhg3XDtol8ifJl3lewsy+l+RC4CyYDjQSkV3DAhHZHDg21zdSi3gYjS3tnIJFO7wa\nbSMijRP6rMCWGOpE2jQUkR1FJLoEky2ZPEOwH04XRtrWwqIufsccY8H8pbYSkZ6JnUWkbqBwVTTP\nY7IXSw0vIjUDCxKYk+06LIonSp807jEBU9Auj4yXjD+D+5b4/IKlzneAY0QkXG5CRJph0TefqGpJ\nS7N5QaHMwzHcclJ9iTO/quq/KksQoDbmQPg85kNwEfaP5I2g/lPM8vCMxFKkn07yX4bjgZNF5D4s\nRHZ5MM4zWHjnwCDnwSdYuHAXYJCq5kopS2XW/g9wF/BKMIf62Jfvd1jUVK55DngW+/IfneDgCzBV\nRD7E3q+FwF6YM2c0Bf1xWJTR2dj7Vxp7iki/JOUfkNkzBIvK6Ssif8N8CE4BdgN6Rixf/yIWYnwQ\nMBZTanfGlrIOJabIVAiq+rGIPA5cIyK7Y1+WazHr1YlYSPjIwLfk3qDdG5givgdwGMmXfv76XKmq\nishFwGvAJLGcJr9hfzu
      "text/plain": [
       "<matplotlib.figure.Figure at 0x146068c5a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_path = os.path.join('data', 'CIFAR-10')\n",
    "reader_train = create_reader(os.path.join(data_path, 'train_map.txt'), os.path.join(data_path, 'CIFAR-10_mean.xml'), True)\n",
    "reader_test  = create_reader(os.path.join(data_path, 'test_map.txt'), os.path.join(data_path, 'CIFAR-10_mean.xml'), False)\n",
    "\n",
    "pred = train_and_evaluate(reader_train, reader_test, max_epochs=5, model_func=create_basic_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although, this model is very simple, it still has too much code, we can do better. Here the same model in more terse format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_basic_model_terse(input, out_dims):\n",
    "\n",
    "    with default_options(activation=relu):\n",
    "        model = Sequential([\n",
    "            For(range(3), lambda i: [\n",
    "                Convolution((5,5), [32,32,64][i], init=glorot_uniform(), pad=True),\n",
    "                MaxPooling((3,3), strides=(2,2))\n",
    "            ]),\n",
    "            Dense(64, init=glorot_uniform()),\n",
    "            Dense(out_dims, init=glorot_uniform(), activation=None)\n",
    "        ])\n",
    "\n",
    "    return model(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 116906 parameters in 10 parameter tensors.\n",
      "\n",
      "Finished Epoch[1 of 300]: [Training] loss = 2.054147 * 50000, metric = 75.0% * 50000 13.674s (3656.6 samples per second);\n",
      "Finished Epoch[2 of 300]: [Training] loss = 1.695077 * 50000, metric = 62.6% * 50000 14.271s (3503.7 samples per second);\n",
      "Finished Epoch[3 of 300]: [Training] loss = 1.542115 * 50000, metric = 56.3% * 50000 13.872s (3604.3 samples per second);\n",
      "Finished Epoch[4 of 300]: [Training] loss = 1.450798 * 50000, metric = 52.3% * 50000 13.823s (3617.3 samples per second);\n",
      "Finished Epoch[5 of 300]: [Training] loss = 1.373555 * 50000, metric = 49.2% * 50000 13.857s (3608.4 samples per second);\n",
      "Finished Epoch[6 of 300]: [Training] loss = 1.300828 * 50000, metric = 46.6% * 50000 13.965s (3580.3 samples per second);\n",
      "Finished Epoch[7 of 300]: [Training] loss = 1.232516 * 50000, metric = 43.7% * 50000 13.827s (3616.0 samples per second);\n",
      "Finished Epoch[8 of 300]: [Training] loss = 1.189415 * 50000, metric = 42.0% * 50000 13.885s (3600.9 samples per second);\n",
      "Finished Epoch[9 of 300]: [Training] loss = 1.134052 * 50000, metric = 39.9% * 50000 13.871s (3604.6 samples per second);\n",
      "Finished Epoch[10 of 300]: [Training] loss = 1.098405 * 50000, metric = 38.9% * 50000 13.961s (3581.3 samples per second);\n",
      "\n",
      "Final Results: Minibatch[1-626]: errs = 36.2% * 10000\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xd4VGX2wPHvARQVBRRWQBcQBBVEUdC1YndtKOpPhWBn\nXRV0dVnXXtFVdl0FbNhBLETQtYAFsSt2g3VFcEFUREQBg3RIzu+Pc6/3zmSSTMpkJpPzeZ557tz3\ntvedCeTkraKqOOecc87likbZzoBzzjnnXJwHJ84555zLKR6cOOeccy6neHDinHPOuZziwYlzzjnn\ncooHJ84555zLKR6cOOeccy6neHDinHPOuZziwYlzzjnncooHJy4viEipiFxVzWvnisiY2P6pwf16\n1V4Oq09EOgb5+Vu28+LKJyLvishz1bz2URGZUdt5SvPZ1c63c5niwYnLGbGgoFRE9iznnO+C45OS\nDmnwqo7SFNfW6roOIlIgIufX5j1d+URkbOxnqaLXmMrvlraa/Awq9nOYDb6Gics5TbKdAedSWAkM\nBN6OJ4rIvsCWwKoU12wIrKvm87Yl878YBgLbA7dk+DnO3AW8GNvvBFwL3AO8GUufXYvP7EP1f9Gf\nBEgt5sW5es2DE5eLngOOF5HzVDUeNAwEPgRaJ1+gqmuq+zBVXVvda+srEdlIVVdkOx+ZoqrvAe+F\n+yLSG7gOeEdVx6dzDxHZQFVTBcLlPbO6wTGqWlLda53LR96s43KNAoVAK+DgMFFE1gOOA8aT4i/M\n5D4nInJNkLa1iDwgIktE5BcRGSMiGyRdO7ec6v1mInK3iPwsIsUiMk5EWiZde5SIPCMi34vIKhH5\nn4hcISKNYue8ChwBhH1HSkVkTux40yC/M0VkpYjMF5H/iEinFOX8c/CMVSLyvojsUtkHGmsu20dE\nRovIj8B3wbEHROTrFNdcIyKlSWmlInKriPQTkc+CPHwuIodU8vzNRWStiFyZ4tg2wX2HBPtNRORq\nEZkVfBY/i8ibInJgZeWsCRFZICITReQIESkSkVXAKcGxP4vIKyLyY5Cnz0RkUIp7JPTdEJFDgrId\nFXye34vIChF5QUQ6Jl2b0OdERLYNP5fgNTt49tsi0jPFsweKyIzgnI+DclS7H4uItAl+NhYG9/xI\nRApSnHeKiEwXkV+Df1+fiMjg2PH1ReQfIvJVcJ+fROR1EdmnOvlyDYfXnLhcNBd4FygAXgjSDgea\nA48C6fTdCKvXJwJzgEuAXsAZwI/ApSnOjRPgdmAJcDXW9DME6ADsHzvvNOBX4GZgGXAA1nywCXBx\ncM4/gBZYk9Rfg3svAwiCmGeDexYCo4JrDwZ6APHA4URgY6zJQoP7/0dEOqf5l/doYCEwDNgoVvZU\n5S8vvQ9wbHCvX4HzgMdFpIOqLkn1UFVdKCKvAydgtRdxA7DmuInB/jDsu7oH+AD7znfBvruXKy9i\ntSmwIzAOK9tdwH+DY0OCvDyJNf8dDdwnIqqqY5PukcrVwGrgn1jQfRHwAIk/R+V93n8CNgDuABpj\n3/njIrKNqiqAiBwLPIzVKl6M1Sw+BMyvIE/lEpFmwDTs5/VWYB7QH3hERDZW1XuD844MyvE8cDf2\nx+72wB7AncHthmP/Xu8EPsL+HfwB2Al4o6p5cw2IqvrLXznxAk4FSrBfREOAX4CmwbEJwEvB+6+B\nSUnXlgJXxfavDtLuSTrvP8DCpLSvgTFJ+SjFmgUax9L/HuSvbyytaYpy3In94l4vljYZmJPi3NOD\nZ51XwefSMThnIdA8ln5kkJ/D0/hcS4HXAEk6NracfF0NlKT4jFcCW8XSdgjSh1SShz8Hee2elP45\n8GJs/6Pk77aWfrZ6B/k8pZzjPwT52zvFsVTf8SvAZ0lp7wDPxfYPCZ45Penn6MLgWZ1jaYXAF7H9\nsB/U90CzWPrxwbUHxNJmAl/F84kFt6Xxe1bw2STn++LgGUfH0ppgwc8iYIPYz/mCSu49A5hY29+n\nv/L/5c06LldNxP667ysiGwN9gUeqeA/F/qKLexNoFdyzMvdoYo3EnQTBwG8PUF0dvheRjUWkFfZX\n50bAdmk841jgJ6yWpjKPqurS2P6bWC1M5zSuVeBeVa3pyIwXVXXubzdV/QxYmkYensA+u/5hgohs\nD3THasNCvwDbi0iXGuazOmao6rTkxKTvuIWItMb+6u8mIuuncd/7kn6Owg656Xxvj6jq8qRrf/vO\ng6a/rsDYeD5V9UUsYKmOw4BvVPWp2P3WAbcBLYFwJN0vQAsROaCCe/0C7JiqidK5inhw4nKSqv4M\nvIR1gj0W+1l9vBq3+jZpP2x62LSyLAD/S8rTcuwv7K3CNBHpLiJPisgv2C/pn7AqdbAq7MpsDczU\nxI6/5fkuKT+/BG8rK0tobprnpZ2HwJLK8qCqi7BmmRNiyQOAtVhzSegq7BfgLBH5VERuFJEdapbl\ntJXpewM2SkxEXhWR5VhZFwb5FKzZqTLJn9mS4Np0vrdU1xK7Nuy7kmrU0f9SpKWjIzArRfoMLN/h\nM28DvgFeFJFvROReETko6ZrLgTbA7KAvzHAR6V7NfLkGxIMTl8vGY7UUZwPPq+qv1bhHeX0xajxs\nU0RaYH9B7wBcgdXuHETU16S2/33VtCwrU6SVV5PSOAN5eBTYRkR2DPaPB15W1cW/ZUb1TSxgOx34\nDOtzMT1VB9QMKPP5iMh2wFSgGdZ34nDsOw5rutL5jmvymWXs57emVHU+9rN/DNZv6iBgqojcGTvn\nFez7/BMW3JwFfCwiJ9Z9jl194sGJy2VhB8TdsEClLglWXR4lWEfBdkQ1EPthf8Geqqq3q+pzwX/G\nv1BWeUHAbGBbESkvGMi0JVhNRbKtMvCsp7Cakv7BiJNtsL4WCVT1F1Udp6onAu2BT4FrMpCfdPTD\n+lscrqr3qeqU4DvOleHn3wTbVM1g1W0a+wb7bpJ1w36Ow2eiqmtVdZKqDsGamh4AzhSRLWLnLFbV\nsapagHUon4n1aXKuXB6cuJwVNKOcjf1impyFLJwpIvERbUOwGoVwuGgJFsTEhw2vH5yXbDmpm3n+\nA/wOOLc2MlwNs7F+Az3CBBFph41IqVWqWoyNvjoBa9JZDTwdP0dENku6ZgXWPNE0dk7zYKhtOk0q\nNRXWXMS/41bYpGnpyOjsq6r6Nda35DSJDZEXG97dtdwLK/YcNuy9X+x+TbCf0V+At4K05O9KsQ7O\nEHxfKc5Zho2ea4pzFfChxC7XJFRXq+pD5Z1YB9YHXhaRiVjn1sHAm6r6THD8bazm4UERuTVIO4nU\nv5CKgBNE5GZsWOqy4D4PYvNpjBCR3bAOjxsDBwJ3qGptBWXlNQM8CvwLeCooQzMsIJyJjZqqbROw\nYa9DgBeSOvgCfCEir2Gf12JgV2x+m1tj5xyDjTI6Dfv8MmkKcAPwvIjch9UynYmNoikzGWAKddH8\ncjn2uU4TkQeBzbGf1f9SvT9A78CG3I8Xkduxfi8DsJ+Hs2Mdbx8WkabYKLDvsZqTc4H3gqAJrK/J\n89iIpSXYMOO+wI3VyJdrQDw4cbkmnb80U80JUdN1TVLd71xsbpFhwHrYaKHf5lhR1cUicgQ2x8l1\n2H++D2HDTF9Iut9ooCf2C/WvWNX4M6paKiKHYb9gws6/i7Ag5bM0ypduuVOeE5ThaGAEFqR8jc0z\nsg1lg5Oa5gFgEta3oxmJo3RCtwBHYUNhm2Kf02XATSmeWVUVXZOyDKr6uYgcj32/N2O/hEditT6j\n03hGec8s73NM99rfjqnq4yJyMnAl9h3OxILkIcAWKe9QwbNVdbmI9MHmZTkdm3dnBnCiqsa/swew\nviRDsKDtByxYHBY7ZyQ2AeEh2Pf5NTaUelSa+XINlNR8ZKFzzrlcE8wOO0tV+1V6snM5Jut9TkTk\nUrFpuJeKTQ/9pIik6oxV3vV7iU2NPT2T+XTOuVwkNuV/o6S0Q7GJ3F7NTq6cq5ms15yIrUVRiM0+\n2ASb7rgH0E1VUw19jF/
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14606872668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzsnWeYFFXWgN8DIkkFEQEzilkMgGGNqOhiQkVdcTC7ZlaR\nNa0ZxV3DKhhRjIAoqKufYkRBUQQUHZIoGJGgBFFAwhDnfD9OlV3d0z3T3fTM9DTnfZ56qurWrVvn\ndvdMnz73BFFVHMdxHMdx8oVa1S2A4ziO4zhOFFdOHMdxHMfJK1w5cRzHcRwnr3DlxHEcx3GcvMKV\nE8dxHMdx8gpXThzHcRzHyStcOXEcx3EcJ69w5cRxHMdxnLzClRPHcRzHcfIKV06cjBCRUhG5Nct7\nfxKRZyLn5wbjtc2dhNkjItsF8vyzumVZnxGRkSIyOcdjxn328pkkfyftg8/lYTl8RtZ/x45TFbhy\nsh4SUQpKReSgFH1mBdeHJlzSYMuG0iT35rR+gogUiUj3XI7pVEzwWXkoR8NVRk2NtMaM/F2Uisha\nEflZRIaJSPtKkCkVyWTN+DURkWNF5LZynlHltUtE5LaE1zjx9W5W1TI5+ckG1S2AU62UAF2BMdHG\n4B/xVsCKJPfUB9Zk+bxdMAWlMukK7AE8WMnPcQqX94CBgADbA5cDH4jIcao6rKqFUdWPRKS+qq7K\n8NbjMNlvT3JtXf6O1xUFLgWWJbm2qIplcfIUV07Wb94G/iYiV6pqVGnoCnwBNE28IYt/kNF7V2d7\nb01FRBqo6vLqlsPJiG9V9YXwREReAyYDVwFJlRMREWBDVV1ZGQJl+XcnOR4vl7yiqr9ncoOI1AVW\naZJqtbn4O/O/1fzCl3XWXxQYDGwGHB02ikgd4DTgBZL8c0tcqxaRnkFbKxHpLyILRWSRiDwjIvUS\n7k217t9QRPqJyAIRWSwiA0SkccK9J4rIm4GZfYWIfC8iN4tIrUifD4HjgdB3pFREfoxcrxvI+42I\nlIjILyLyiohsn2SeFwXPWCEi40Rk34pe0Mhy2WEi0ldE5gGzgmv9RWR6knt6ikhpQlupiDwkIieJ\nyJeBDFNEpGMFz28mIqtF5JYk13YOxr08ON8gMLF/G7wWC0RklIh0qGie2ZLOe5jQv62IjBaR5SLy\no4hckqTPhiJyu4h8F4w5U0TuEZENcyW3qk4BFmBWlPC54XvUVUSmYFbGjsE1EZGrgvesRETmisjj\niZ/poO/NYkuoy0RkhIjsnqRPUp8TETlARN4Wkd9FZKmITBKRK4Jrz2JWk+hS1doE+W9NGK+NiLwT\n/A0uEZHhInJAQp/wM36QiPQWkfnBs18Vkc0yfnFTEJlzFxG5U0RmY5aWjUXkvFR/Z1nMI+kYTvXj\nlpP1m5+AT4EiYr8IjwM2AYYA6fhuhL9iXgJ+BP4FtAUuBOYBNyTpG0WAR4CFwG3Y0s/lwLbAEZF+\n5wFLgPuBpcCRwB3AxsD1QZ87gUbYktRVwdhLAYIvwLeCMQcDDwT3Hg20BqKKw5nARsDjgczXA6+I\nyA6qupaK6QvMx8zpDSJzT+VLkKz9UOCUYKwlwJXA/0RkW1VdmOyhqjpfRD4CTgd6JVw+AzPjvxSc\n3469V08An2Pv+b7Yezei4ilmxXlU/B6GNMHer5cwRfl04DERWamq/eFPa8UbwEFAP2AasCfQA9gJ\ne/3WGRHZFNgU+C7hUodArkcw5eWnoP0J4BzgGWx5cXvgCmAfETk4/AyJSC/gJuBN4B3stX8PqJNE\njLjPiIgcjc39F+yzPBfYDTgBeBh7PbYEjsI+zymtKMF4uwMfA4uBu7HPyiXASBE5TFU/T7jlYeB3\noCfQEnvNH8H+l6TDZsH7F2WNqi5OaLsFWAn8F6gLrCL2WkT/zhoG89gjw3mUGcPJE1TVt/VsA84F\n1mL/DC/H1nnrBtdeBIYHx9OBoQn3lgK3Rs5vC9qeSOj3CjA/oW068EyCHKXAZ0DtSPs1gXwnRNrq\nJpnHY9iXXZ1I2xvAj0n6nh8868pyXpftgj7zgU0i7Z0CeY5L43UtBUYCknDt2RRy3QasTfIalwAt\nI217Bu2XVyDDRYGsuye0TwHej5xPSHxv1/EzVQo8VEGfdN/DD4M5dI+01QHGA3PCzwpwFrAaODBh\nzIuD+/+S6rNXwTyewCyKTYH9geFJ5CkNnr1Lwv2HBNe6JLQfHbSfEZw3xawtryf0uzPoF/07aR88\n/7DgvBb2Q+AHYONy5vJw4mcrQf7o3/H/BZ+57SJtLbAv+Q+TfMbfTRjvfkxxSClP5PNemmL7OmHO\npZhCuGEGf2eZzqPMGL7lx+bLOs5L2K/7E0RkI+yX1/MZjqHYL7Uoo7BfRxulcf8TGm+ReIxAGfjz\nAZG1fBHZKDAhfxLIvmsazzgF+BX7dVcRQ1T1j8j5KOyX5w5p3KvAkxr8B1wH3lfVn/4cVPVL4I80\nZHgVe+26hA3Br8ndMWtYyCJgDxHZcR3lTJsM38M1mJIQ3rsa+4w1A9oFzacBU4FvRWSzcMOUGyHe\n8pYJf8c+K/Mxy+KBwP2qmuhkPVJVv0loOw17bUckyDQBsxaFMh2NKVwPJ9z/QBrytcGsFQ+o6pL0\nppSawKp4NPB/qjojbFfVuZjV6pCEv2Ml8t4EjAJqYwp+RSjQGbPqRLfzk/Ttr8n9Y8r8nWU5j1z8\nrTqVgC/rrOeo6gIRGY45wTbEfpX9L4uhZiach0sPmxIsraQSAfg+QaZlIjIH+wcM/Gl2/jf2z32T\nhPsbpSFfK+AbjXf8TUXc2rOqLgos0JumcS/EzPvrQrL174UVyaCqv4nICGy5IQwjPQP7lf9/ka63\nAq9hX+xTgHeB5wIlqFLI8D38RVVLEtq+xZSOlsA4bOlmV0yRSEQxRSYbXseUWMWsOl8lkQWSv887\nAY0xxaY8mbYN9omf/QUiknTZLkKrYKyvKuiXLptjCuK3Sa5Nxf4nbBMchyR+PqN/7+kwStNziP0p\ng2vZzKO88Z1qxJUTB+xXxZPAFsA7Wf4aS+WLUe5adzqISCNsHXkRcDNm0l6B/YK+m9w7dq/rXJJ9\nkaX6dVa7EmQYAjwjInup6mTgb8CI6JeBqo4SkVbAScBfMWtBDxG5RFVznqyskt7DWsCXmL9Dstcl\nWwfH2ar6QRr9kr3PtTBfq64pZEqmSNVEKu3vPYFkr3E613IxvlONuHLigP2i7gccQGQ5oIoQ7Nfm\nR382iDTEFKW3gqbDsV9kJ6nq6Ei/VknGS6UE/ADsLyK1NT2n1lyzEPtFnUjLSnjWa9j72SVwOtwZ\ns1jEoaqLgAHAABFpgJnme2KOnLnmcNJ/DwG2FMvtEf3y2AV7f0Pn5R+AvVT1w0qQN1t+wBxlx2j5\nYcXhssNORH69i0hTKrY+/ID93bQGylOi0l2u+BVYjr2+ieyG+WbUhEiWQpmHg4cSO9gyCpYUqSfm\nUFrVXCwiUUX5csyi8HZwvhb7ZxwNG94w6JfIMpIv87yCmX3/kQuBs+AHoJGItA4bRGQL4ORcP0gt\n4mEYtrRzBhbt8Hq0j4g0SbhnObbEUDfSZxMR2UVEoksw2ZLJewj2w+nSSN86WNTFr5hjLJi/1NYi\nclHizSJSL1C4qpqXMNnLpIYXkdqBBQnMyXYNFsUTpUcazxiPKWhXRcZLxrLgueW+f8FS53vASSIS\nLjchIs2x6JtRqlre0mxeUCjzcAy3nKy/xJlfVfW56hIE2BBzIHwJ8yG4DPtH8mZwfQxmeRgosRTp\nZ5H8l2ExcLqI3I+FyC4NxhmIhXf2DnIejMLChTsAj6pqrpSyVGbtIcA9wGvBHBpiX77fYFFTueZF\nYBD25T8swcEX4GsRGYm9Xr8D+2HOnNEU9J2xKKPzsNevIvYVkZuStH9IZu8hWFTOdSLSEvMhOAPY\nC7goYvl6jliI8RHAaEyp3Q1byvorMUWmSlDVj0WkH/AvEdkH+7JcjVmvTsNCwl8NfEvuC/q9iSni\nbYBjSL708+fnSlVVRC4
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14606eb3f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred_basic_model = train_and_evaluate(reader_train, reader_test, max_epochs=10, model_func=create_basic_model_terse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a trained model, let's classify the following image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/201/00014.png\" width=\"64\" height=\"64\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 6\n",
    "Image(url=\"https://cntk.ai/jup/201/00014.png\", width=64, height=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "\n",
    "def eval(pred_op, image_path):\n",
    "    label_lookup = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\n",
    "    image_mean   = 133.0\n",
    "    image_data   = np.array(Image.open(image_path), dtype=np.float32)\n",
    "    image_data  -= image_mean\n",
    "    image_data   = np.ascontiguousarray(np.transpose(image_data, (2, 0, 1)))\n",
    "    \n",
    "    result       = np.squeeze(pred_op.eval({pred_op.arguments[0]:[image_data]}))\n",
    "    \n",
    "    # Return top 3 results:\n",
    "    top_count = 3\n",
    "    result_indices = (-np.array(result)).argsort()[:top_count]\n",
    "\n",
    "    print(\"Top 3 predictions:\")\n",
    "    for i in range(top_count):\n",
    "        print(\"\\tLabel: {:10s}, confidence: {:.2f}%\".format(label_lookup[result_indices[i]], result[result_indices[i]] * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 3 predictions:\n",
      "\tLabel: truck     , confidence: 98.95%\n",
      "\tLabel: ship      , confidence: 0.46%\n",
      "\tLabel: automobile, confidence: 0.26%\n"
     ]
    }
   ],
   "source": [
    "eval(pred_basic_model, \"data/CIFAR-10/test/00014.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding dropout layer, with drop rate of 0.25, before the last dense layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_basic_model_with_dropout(input, out_dims):\n",
    "\n",
    "    with default_options(activation=relu):\n",
    "        model = Sequential([\n",
    "            For(range(3), lambda i: [\n",
    "                Convolution((5,5), [32,32,64][i], init=glorot_uniform(), pad=True),\n",
    "                MaxPooling((3,3), strides=(2,2))\n",
    "            ]),\n",
    "            Dense(64, init=glorot_uniform()),\n",
    "            Dropout(0.25),\n",
    "            Dense(out_dims, init=glorot_uniform(), activation=None)\n",
    "        ])\n",
    "\n",
    "    return model(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 116906 parameters in 10 parameter tensors.\n",
      "\n",
      "Finished Epoch[1 of 300]: [Training] loss = 2.123667 * 50000, metric = 78.7% * 50000 16.391s (3050.5 samples per second);\n",
      "Finished Epoch[2 of 300]: [Training] loss = 1.817045 * 50000, metric = 67.9% * 50000 16.894s (2959.5 samples per second);\n",
      "Finished Epoch[3 of 300]: [Training] loss = 1.678272 * 50000, metric = 62.2% * 50000 17.006s (2940.1 samples per second);\n",
      "Finished Epoch[4 of 300]: [Training] loss = 1.583182 * 50000, metric = 58.1% * 50000 16.644s (3004.1 samples per second);\n",
      "Finished Epoch[5 of 300]: [Training] loss = 1.514311 * 50000, metric = 55.3% * 50000 16.790s (2977.9 samples per second);\n",
      "\n",
      "Final Results: Minibatch[1-626]: errs = 49.2% * 10000\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYVOXZx/HvDSooRBALiFFEBUVQDKtRo1jQiB2NsWBv\nMQajBn3tGk2x94axYkeNLfYWe4kFNKKosQCiNBVBaQrs/f5xn3HOzs7uzi67O7Ozv891zTXnPKfM\n8+xZmHufau6OiIiISKloU+wMiIiIiKQpOBEREZGSouBERERESoqCExERESkpCk5ERESkpCg4ERER\nkZKi4ERERERKioITERERKSkKTkRERKSkKDiRsmBmlWb25wZeO8HMbkrtH5Tcb0Dj5bDhzKxHkp/j\nip0XqZmZ/cfMHmvgtXeZ2QeNnacCP7vB+RZpKgpOpGSkgoJKM/tVDedMSo4/lHPIk1dDVOa5tlHX\ndTCzoWZ2bGPeU2pmZiNTv0u1vW6q+24FW5zfQSd+D4tBa5hIyVmi2BkQyWMesC/wajrRzLYEVgHm\n57lmaWBhAz9vbZr+i2FfoC9weRN/joR/AE+n9nsCfwWuA15KpX/aiJ85kIZ/0e8PWCPmRaRFU3Ai\npegxYE8zO8bd00HDvsBbwAq5F7j7jw39MHdf0NBrWyozW8bd5xY7H03F3V8HXs/sm1kF8DfgNXe/\ns5B7mFl7d88XCNf0mQ0NjnH3RQ29VqQcqVlHSo0Do4DlgV9nEs1sSeC3wJ3k+Qszt8+JmZ2VpK1p\nZjeb2bdmNtPMbjKz9jnXTqiher+DmV1rZl+b2Swzu8XMOudcu6uZPWJmX5rZfDP7xMxON7M2qXOe\nA3YCMn1HKs3ss9Txdkl+PzKzeWY22czuM7Oeecr5u+Qz5pvZG2a2YV0/0FRz2RZmNsLMpgGTkmM3\nm9n4PNecZWaVOWmVZnaFmQ0xs7FJHt4zs8F1fP5KZrbAzM7Ic6x3ct9hyf4SZnammf0v+Vl8bWYv\nmdk2dZVzcZjZVDO7x8x2MrPRZjYfODA59jsze9bMpiV5Gmtmh+a5R5W+G2Y2OCnbrsnP80szm2tm\nT5pZj5xrq/Q5MbO1Mz+X5PVp8tmvmln/PJ+9r5l9kJzzTlKOBvdjMbOuye/G9OSeb5vZ0DznHWhm\nY8zs++Tf13/N7A+p40uZ2d/N7OPkPl+Z2QtmtkVD8iWth2pOpBRNAP4DDAWeTNJ2BJYF7gIK6buR\nqV6/B/gMOBkYABwOTANOyXNumgFXAd8CZxJNP8OA1YCtU+cdDHwPXAzMBgYRzQc/A05Kzvk70Ilo\nkvpTcu/ZAEkQ82hyz1HAZcm1vwb6AenAYT+gI9Fk4cn97zOzNQr8y3sEMB34C7BMquz5yl9T+kDg\nN8m9vgeOAe41s9Xc/dt8H+ru083sBWAvovYibR+iOe6eZP8vxLO6DniTeOYbEs/u33UXscEcWB+4\nhSjbP4D3k2PDkrw8QDT/7QbcYGbu7iNz7pHPmcAPwHlE0H0icDNVf49q+nkfBrQHrgbaEs/8XjPr\n7e4OYGa/AW4nahVPImoWbwMm15KnGplZB+Bl4vf1CuALYG/gDjPr6O7XJ+ftkpTjceBa4o/dvsCm\nwDXJ7c4l/r1eA7xN/Dv4JbAB8GJ98yatiLvrpVdJvICDgEXEF9EwYCbQLjl2N/BMsj0eeCjn2krg\nz6n9M5O063LOuw+YnpM2HrgpJx+VRLNA21T6/yX52zmV1i5POa4hvriXTKU9DHyW59xDks86ppaf\nS4/knOnAsqn0XZL87FjAz7USeB6wnGMja8jXmcCiPD/jecDqqbT1kvRhdeThd0le181Jfw94OrX/\ndu6zbaTfrYoknwfWcHxKkr/N8xzL94yfBcbmpL0GPJbaH5x85pic36MTks9aI5U2ChiX2s/0g/oS\n6JBK3zO5dlAq7SPg43Q+ieC2Mn3PWn42ufk+KfmM3VJpSxDBzzdA+9Tv+dQ67v0BcE9jP0+9yv+l\nZh0pVfcQf93vbGYdgZ2BO+p5Dyf+okt7CVg+uWddrvOqNRLXkAQDP32A+w+ZbTPraGbLE391LgOs\nU8Bn/Ab4iqilqctd7v5dav8lohZmjQKudeB6d1/ckRlPu/uEn27qPhb4roA83E/87PbOJJhZX2Bd\nojYsYybQ18zWWsx8NsQH7v5ybmLOM+5kZisQf/X3MbOlCrjvDTm/R5kOuYU8tzvcfU7OtT8986Tp\nrxcwMp1Pd3+aCFgaYgdgors/mLrfQuBKoDOQGUk3E+hkZoNquddMYP18TZQitVFwIiXJ3b8GniE6\nwf6G+F29twG3+jxnP9P0sFxdWQA+ycnTHOIv7NUzaWa2rpk9YGYziS/pr4gqdYgq7LqsCXzkVTv+\n1mRSTn5mJpt1lSVjQoHnFZyHxLd15cHdvyGaZfZKJe8DLCCaSzL+THwB/s/M3jWzC8xsvcXLcsGq\n9b2BGCVmZs+Z2RyirNOTfBrR7FSX3J/Zt8m1hTy3fNeSujbTdyXfqKNP8qQVogfwvzzpHxD5znzm\nlcBE4Gkzm2hm15vZtjnXnAZ0BT5N+sKca2brNjBf0oooOJFSdidRS3Ek8Li7f9+Ae9TUF2Oxh22a\nWSfiL+j1gNOJ2p1tyfY1aex/X4tblnl50mqqSWnbBHm4C+htZusn+3sC/3b3GT9lxv0lImA7BBhL\n9LkYk68DahOo9vMxs3WAp4AORN+JHYlnnKnpKuQZL87PrMl+fxeXu08mfvd3J/pNbQs8ZWbXpM55\nlniehxHBze+Bd8xsv+bPsbQkCk6klGU6IG5MBCrNyYjq8mxCdBRcmWwNxFbEX7AHuftV7v5Y8p/x\nTKqrKQj4FFjbzGoKBprat0RNRa7Vm+CzHiRqSvZORpz0JvpaVOHuM939FnffD1gVeBc4qwnyU4gh\nRH+LHd39Bnd/InnGpTL8fGLynq8ZrKFNYxOJZ5OrD/F7nPlM3H2Buz/k7sOIpqabgSPMrHvqnBnu\nPtLdhxIdyj8i+jSJ1EjBiZSspBnlSOKL6eEiZOEIM0uPaBtG1ChkhosuIoKY9LDhpZLzcs0hfzPP\nfcCKwB8bI8MN8CnRb6BfJsHMViZGpDQqd59FjL7ai2jS+QH4V/ocM+uSc81conmiXeqcZZOhtoU0\nqSyuTM1F+hkvT0yaVogmnX3V3ccTfUsOttQQeYvh3b1qvLB2jxHD3oek7rcE8Ts6E3glSct9Vk50\ncIbkeeU5ZzYxeq4dIrXQUGIpNVWqq939tppObAZLAf82s3uIzq1/AF5y90eS468SNQ+3mtkVSdr+\n5P9CGg3sZWYXE8NSZyf3uZWYT+MSM9uY6PDYEdgGuNrdGysoq6kZ4C7gfODBpAwdiIDwI2LUVGO7\nmxj2Ogx4MqeDL8A4M3ue+HnNADYi5re5InXO7sQoo4OJn19TegI4B3jczG4gapmOIEbRVJsMMI/m\naH45jfi5vmxmtwIrEb+r79OwP0CvJobc32lmVxH9XvYhfh+OTHW8vd3M2hGjwL4kak7+CLyeBE0Q\nfU0eJ0YsfUsMM94ZuKAB+ZJWRMGJlJpC/tLMNyfE4q5rku9+fyTmFvkLsCQxWuinOVbcfYaZ7UTM\ncfI34j/f24hhpk/m3G8E0J/4Qv0TUTX+iLtXmtkOxBdMpvPvN0SQMraA8hVa7rznJGXYDbiECFLG\nE/OM9KZ6cLK4eQB4iOjb0YGqo3QyLgd2JYbCtiN+TqcCF+X5zPqq7Zq8ZXD398xsT+L5Xkx8CV9K\n1PqMKOAzavrMmn6OhV770zF3v9fMDgDOIJ7hR0SQPAzonvcOtXy2u88xs4HEvCyHEPPufADs5+7p\nZ3Yz0ZdkGBG0TSGCxb+kzrmUmIBwMPE8xxNDqS8rMF/SStnijywUEZFSk8wO+z93H1LnySIlpuh9\nTszsFItpuL+zmB76ATPL1xmrpus3s5gae0xT5lNEpBRZTPnfJidte2Iit+eKkyuRxVP0mhOLtShG\nEbMPLkFMd9wP6OPu+YY
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14606e63da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJztnWeYFNXSgN8iCwiKqBhAFAOIIsGMYr7oVcSMKBcjmBX8\njNeA13DFiAmzAqKiXiOYUMyCioCCKIooIIpKUpActr4f1e30zM7szszO7s7O1vs8/XT36dPddaYn\n1NSpIKqK4ziO4zhOvlCjsgVwHMdxHMeJ4sqJ4ziO4zh5hSsnjuM4juPkFa6cOI7jOI6TV7hy4jiO\n4zhOXuHKieM4juM4eYUrJ47jOI7j5BWunDiO4ziOk1e4cuI4juM4Tl7hyomTESJSJCLXZnnuLBF5\nPLJ/SnC9jrmTMHtEZKtAnosrW5bqjIi8LyJTcnzNuPdePpPkc7Jf8L7sksN7ZP05dpyKwJWTakhE\nKSgSkb1T9JkTHB+ZcEiDJRuKkpyb0/oJItJTRC7K5TWd0gneK/fk6HLlUVMjrWtGPhdFIrJORH4R\nkdEisl85yJSKZLJm/JqIyGEiMqCEe1R47RIRGZDwGie+3ptUtExOflKrsgVwKpUVwEnAuGhj8EW8\nBbAyyTnrAWuzvN8OmIJSnpwEtAXuLuf7OIXLW8ATgABbA+cC74rIP1V1dEULo6ofiMh6qro6w1P/\nicn+nyTHyvI5LisKnA0sS3LszwqWxclTXDmp3rwOHC8iF6pqVGk4CZgANE08IYsvyOi5a7I9t6oi\nIvVVdXlly+FkxHRVfTrcEZGXgSlAPyCpciIiAtRR1VXlIVCWnzvJ8fVyyQuquiiTE0SkLrBak1Sr\nzcXnzD+r+YVP61RfFBgBbAQcEjaKSG3gOOBpkny5Jc5Vi8h1QVsrERkqIn+IyJ8i8riI1Es4N9W8\nfwMReUhEFojIYhEZJiIbJJx7pIi8GpjZV4rIDBG5WkRqRPq8BxwOhL4jRSLyY+R43UDe70RkhYjM\nFZEXRGTrJOPsE9xjpYiMF5FdS3tBI9NlXUTkfhH5HZgTHBsqIjOTnHOdiBQltBWJyD0i0l1Evgpk\nmCoiXUu5/yYiskZErklybPvguucG+7UCE/v04LVYICIfichBpY0zW9J5hgn9O4rIWBFZLiI/ishZ\nSfrUEZH/iMj3wTV/EpFbRKROruRW1anAAsyKEt43fEYnichUzMrYNTgmItIveGYrROQ3EXkw8T0d\n9L1abAp1mYi8IyI7JumT1OdERPYQkddFZJGILBWRySJyQXBsCGY1iU5VrUuQ/9qE63UQkTeCz+Bf\nIjJGRPZI6BO+x/cWkTtFZF5w7xdFZKOMX9wURMbcQ0RuFJGfMUvL+iJyaqrPWRbjSHoNp/Jxy0n1\nZhbwKdCT2D/CfwKNgGeAdHw3wn8xzwE/AlcAHYEzgd+BK5P0jSLAfcAfwABs6udcoAVwQKTfqcBf\nwB3AUuBA4HpgfeDyoM+NQGNsSqpfcO2lAMEP4GvBNUcAdwXnHgLsBEQVh5OBhsCDgcyXAy+IyDaq\nuo7SuR+Yh5nT60fGnsqXIFn7vsAxwbX+Ai4EnheRFqr6R7Kbquo8EfkAOAG4IeHwiZgZ/7lg/z/Y\ns3oY+Bx75rtiz+6d0oeYFadS+jMMaYI9r+cwRfkE4AERWaWqQ+Fva8UoYG/gIeBbYGegP7Ad9vqV\nGRHZENgQ+D7h0EGBXPdhysusoP1hoDfwODa9uDVwAdBeRDqH7yERuQG4CngVeAN77d8CaicRI+49\nIiKHYGOfi72XfwPaAEcA92Kvx+bAwdj7OaUVJbjejsCHwGJgIPZeOQt4X0S6qOrnCafcCywCrgNa\nYq/5fdh3STpsFDy/KGtVdXFC2zXAKuA2oC6wmthrEf2cNQjG0TbDcRS7hpMnqKov1WwBTgHWYV+G\n52LzvHWDY88CY4LtmcDIhHOLgGsj+wOCtocT+r0AzEtomwk8niBHEfAZUDPSfkkg3xGRtrpJxvEA\n9mNXO9I2CvgxSd/TgntdWMLrslXQZx7QKNLeLZDnn2m8rkXA+4AkHBuSQq4BwLokr/EKoGWkbeeg\n/dxSZOgTyLpjQvtU4O3I/heJz7aM76ki4J5S+qT7DN8LxnBRpK02MAn4NXyvAL2ANcBeCdfsG5y/\nZ6r3XinjeBizKDYFdgfGJJGnKLj3Dgnn7xMc65HQfkjQfmKw3xSztryS0O/GoF/0c7JfcP8uwX4N\n7I/AD8D6JYzl3sT3VoL80c/xS8F7bqtIWzPsR/69JO/xNxOudwemOKSUJ/J+L0qxfJMw5iJMIayT\nwecs03EUu4Yv+bH4tI7zHPbv/ggRaYj983oqw2so9k8tykfYv6OGaZz/sMZbJB4gUAb+vkFkLl9E\nGgYm5I8D2VuncY9jgPnYv7vSeEZVl0T2P8L+eW6TxrkKPKLBN2AZeFtVZ/19UdWvgCVpyPAi9tr1\nCBuCf5M7YtawkD+BtiKybRnlTJsMn+FaTEkIz12Dvcc2AToFzccB04DpIrJRuGDKjRBvecuEM7D3\nyjzMsrgXcIeqJjpZv6+q3yW0HYe9tu8kyPQFZi0KZToEU7juTTj/rjTk64BZK+5S1b/SG1JqAqvi\nIcBLqjo7bFfV3zCr1T4Jn2Ml8mwCPgJqYgp+aShwNGbViS6nJek7VJP7xxT7nGU5jlx8Vp1ywKd1\nqjmqukBExmBOsA2wf2XPZ3GpnxL2w6mHDQmmVlKJAMxIkGmZiPyKfQEDf5udb8K+3BslnN84Dfla\nAd9pvONvKuLmnlX1z8ACvWEa50LMvF8Wks1//1GaDKq6UETewaYbwjDSE7F/+S9Ful4LvIz9sE8F\n3gSGB0pQuZDhM5yrqisS2qZjSkdLYDw2ddMaUyQSUUyRyYZXMCVWMavO10lkgeTPeTtgA0yxKUmm\nFsE68b2/QESSTttFaBVc6+tS+qXLxpiCOD3JsWnYd0LzYDsk8f0Z/bynw0eankPsrAyOZTOOkq7v\nVCKunDhg/yoeATYD3sjy31gqX4wS57rTQUQaY/PIfwJXYybtldg/6IHk3rG7rGNJ9kOW6t9ZzXKQ\n4RngcRFpp6pTgOOBd6I/Bqr6kYi0AroD/8CsBf1F5CxVzXmysnJ6hjWArzB/h2SvS7YOjj+r6rtp\n9Ev2nGtgvlYnpZApmSJVFSm3z3sCyV7jdI7l4vpOJeLKiQP2j/ohYA8i0wEVhGD/Nj/4u0GkAaYo\nvRY07Y/9I+uuqmMj/VoluV4qJeAHYHcRqanpObXmmj+wf9SJtCyHe72MPc8egdPh9pjFIg5V/RMY\nBgwTkfqYaf46zJEz1+xP+s8QYHOx3B7RH48dsOcbOi//ALRT1ffKQd5s+QFzlB2nJYcVh9MO2xH5\n9y4iTSnd+vAD9rnZCShJiUp3umI+sBx7fRNpg/lmVIVIlkIZh4OHEjvYNAqWFOk6zKG0oukrIlFF\n+VzMovB6sL8O+zKOhg3XCfolsozk0zwvYGbf83MhcBb8ADQWkZ3CBhHZDDgq1zdSi3gYjU3tnIhF\nO7wS7SMiTRLOWY5NMdSN9GkkIjuISHQKJlsyeYZgf5zOjvStjUVdzMccY8H8pbYUkT6JJ4tIvUDh\nqmiew2QvlhpeRGoGFiQwJ9u1WBRPlP5p3GMSpqD1i1wvGcuC+5b4/IKpzreA7iISTjchIpti0Tcf\nqWpJU7N5QaGMwzHcclJ9iTO/qurwyhIEqIM5ED6H+RCcg32RvBocH4dZHp6QWIr0XiT/ZzgROEFE\n7sBCZJcG13kCC++8M8h58BEWLnwQMFhVc6WUpTJrPwPcArwcjKEB9uP7HRY1lWueBZ7EfvxHJzj4\nAnwjIu9jr9ciYDfMmTOagv5oLMroVOz1K41dReSqJO3vkdkzBIvKuUxEWmI+BCcC7YA+EcvXcGIh\nxgcAYzGltg02lfUPYopMhaCqH4rIQ8AVItIe+7Fcg1mvjsNCwl8MfEtuD/q9iiniHYBDST718/f7\nSlVVRM4BRgJfiuU0+RX
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14606ebaef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred_basic_model_dropout = train_and_evaluate(reader_train, reader_test, max_epochs=5, model_func=create_basic_model_with_dropout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add batch normalization after each convolution and before the last dense layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_basic_model_with_batch_normalization(input, out_dims):\n",
    "\n",
    "    with default_options(activation=relu):\n",
    "        model = Sequential([\n",
    "            For(range(3), lambda i: [\n",
    "                Convolution((5,5), [32,32,64][i], init=glorot_uniform(), pad=True),\n",
    "                BatchNormalization(map_rank=1),\n",
    "                MaxPooling((3,3), strides=(2,2))\n",
    "            ]),\n",
    "            Dense(64, init=glorot_uniform()),\n",
    "            BatchNormalization(map_rank=1),\n",
    "            Dense(out_dims, init=glorot_uniform(), activation=None)\n",
    "        ])\n",
    "\n",
    "    return model(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 117290 parameters in 18 parameter tensors.\n",
      "\n",
      "Finished Epoch[1 of 300]: [Training] loss = 1.512835 * 50000, metric = 54.1% * 50000 15.499s (3226.1 samples per second);\n",
      "Finished Epoch[2 of 300]: [Training] loss = 1.206524 * 50000, metric = 42.8% * 50000 16.071s (3111.2 samples per second);\n",
      "Finished Epoch[3 of 300]: [Training] loss = 1.087695 * 50000, metric = 38.3% * 50000 16.160s (3094.1 samples per second);\n",
      "Finished Epoch[4 of 300]: [Training] loss = 1.008182 * 50000, metric = 35.4% * 50000 16.057s (3113.8 samples per second);\n",
      "Finished Epoch[5 of 300]: [Training] loss = 0.953168 * 50000, metric = 33.4% * 50000 16.247s (3077.4 samples per second);\n",
      "\n",
      "Final Results: Minibatch[1-626]: errs = 30.8% * 10000\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XecVNX5x/HPFxRUVIgNGxaiKHbBEkXs3QQ0sa29xZ8t\nGmKiMcaoaUZjLFE0drGtGmPBEjv2voiKIBKxISIqsCod9vn98dxx7s7O7s6W2Zndfd6v17xm77nn\n3nvO3IV59rQrMyOEEEIIoVx0KXUBQgghhBDSIjgJIYQQQlmJ4CSEEEIIZSWCkxBCCCGUlQhOQggh\nhFBWIjgJIYQQQlmJ4CSEEEIIZSWCkxBCCCGUlQhOQgghhFBWIjgJHYKkGkl/aOaxH0m6MbV9ZHK+\nAa1XwuaTtGZSnl+VuiyhfpJekfRIM4+9U9L41i5TgddudrlDKJYITkLZSAUFNZK2rSfPp8n+kTm7\nLHk1R02eY1v1uQ6SKiSd1prnDPWTdFPqd6mh142Nn61gLfkdNPz3sBTiGSah7CxW6gKEkMcc4BDg\npXSipB2A1YC5eY5ZEljYzOutR/G/GA4BNgQuL/J1gvsX8ERqe23gj8C1wPOp9A9a8ZqDaf4X/WGA\nWrEsIbRrEZyEcvQIcICkU80sHTQcArwBrJB7gJnNb+7FzGxBc49tryQtZWazS12OYjGzV4FXM9uS\nBgJ/Al42szsKOYekJcwsXyBc3zWbGxxjZouae2wIHVF064RyY0AlsDywWyZR0uLA/sAd5PkLM3fM\niaTzkrQfSrpZ0gxJMyXdKGmJnGM/qqd5v4ekayR9Jala0ghJvXKOHSLpIUmfSZor6X+Sfi+pSyrP\nKGAfIDN2pEbSpNT+7kl5J0iaI2mKpP9IWjtPPX+eXGOupNckbdHYB5rqLtte0lWSvgA+TfbdLOnD\nPMecJ6kmJ61G0j8lDZX0TlKGsZL2aOT6K0laIOmcPPv6Jec9KdleTNK5kt5PPouvJD0vaZfG6tkS\nkqZKulvSPpKqJM0Fjkj2/VzS05K+SMr0jqRj8pyj1tgNSXskdRuSfJ6fSZot6TFJa+YcW2vMiaT1\nMp9L8vogufZLkjbNc+1DJI1P8oxJ6tHscSySeie/G9OSc74pqSJPviMkjZb0bfLv6y1JJ6b2d5P0\nZ0kTk/N8KelZSds3p1yh84iWk1COPgJeASqAx5K0vYFlgTuBQsZuZJrX7wYmAb8FBgDHAV8AZ+XJ\nmybgSmAGcC7e9XMSsAawUyrfUcC3wD+A74Cd8e6DZYAzkzx/BnriXVK/TM79HUASxDycnLMSuCw5\ndjdgIyAdOBwKLI13WVhy/v9I6lvgX95XAdOA84GlUnXPV//60gcDP03O9S1wKnCPpDXMbEa+i5rZ\nNEnPAgfirRdpB+PdcXcn2+fj9+pa4HX8nm+B37unGq9isxmwCTACr9u/gHeTfSclZbkP7/7bF7he\nkpnZTTnnyOdcYB7wNzzoPgO4mdq/R/V93scCSwDDga74Pb9HUj8zMwBJPwVuw1sVz8RbFm8FpjRQ\npnpJ6gG8gP++/hOYDBwE3C5paTO7Lsn3k6Qe/wWuwf/Y3RDYBrg6Od0F+L/Xq4E38X8HWwGbAc81\ntWyhEzGzeMWrLF7AkcAi/IvoJGAm0D3ZdxfwZPLzh8DInGNrgD+kts9N0q7NyfcfYFpO2ofAjTnl\nqMG7Bbqm0n+dlO/HqbTueepxNf7FvXgq7UFgUp68RyfXOrWBz2XNJM80YNlU+k+S8uxdwOdaAzwD\nKGffTfWU61xgUZ7PeA6wVipt4yT9pEbK8POkrBvkpI8Fnkhtv5l7b1vpd2tgUs4j6tn/eVK+7fLs\ny3ePnwbeyUl7GXgktb1Hcs3ROb9Hv0mu1TeVVgmMS21nxkF9BvRIpR+QHLtzKm0CMDFdTjy4rUmf\ns4HPJrfcZybX2DeVthge/HwNLJH6PZ/ayLnHA3e39v2MV8d/RbdOKFd343/d/1jS0sCPgdubeA7D\n/6JLex5YPjlnY6612i0SV5MEA99fwGxe5mdJS0taHv+rcylg/QKu8VPgS7yVpjF3mtk3qe3n8VaY\nvgUca8B1ZtbSmRlPmNlH35/U7B3gmwLKcC/+2R2USZC0IbAB3hqWMRPYUNI6LSxnc4w3sxdyE3Pu\ncU9JK+B/9feX1K2A816f83uUGZBbyH273cxm5Rz7/T1Puv7WBW5Kl9PMnsADlubYC/jYzO5PnW8h\ncAXQC8jMpJsJ9JS0cwPnmglskq+LMoSGRHASypKZfQU8iQ+C/Sn+u3pPM071Sc52puvhB40VAfhf\nTplm4X9hr5VJk7SBpPskzcS/pL/Em9TBm7Ab80NggtUe+FufT3PKMzP5sbG6ZHxUYL6Cy5CY0VgZ\nzOxrvFvmwFTywcACvLsk4w/4F+D7kt6WdJGkjVtW5ILVGXsDPktM0ihJs/C6TkvKKbzbqTG5n9mM\n5NhC7lu+Y0kdmxm7km/W0f/ypBViTeD9POnj8XJnrnkF8DHwhKSPJV0nadecY84GegMfJGNhLpC0\nQTPLFTqRCE5CObsDb6U4AfivmX3bjHPUNxajxdM2JfXE/4LeGPg93rqzK9mxJq3976uldZmTJ62+\nlpSuRSjDnUA/SZsk2wcAT5nZ9O8LY/Y8HrAdDbyDj7kYnW8AahHU+XwkrQ88DvTAx07sjd/jTEtX\nIfe4JZ9Z0X5/W8rMpuC/+/vh46Z2BR6XdHUqz9P4/TwWD27+Dxgj6dC2L3FoTyI4CeUsMwBxazxQ\naUvCm8uzCT5QcBWyLRA74n/BHmlmV5rZI8l/xjOpq74g4ANgPUn1BQPFNgNvqci1VhGudT/eUnJQ\nMuOkHz7WohYzm2lmI8zsUKAP8DZwXhHKU4ih+HiLvc3sejN7NLnH5TL9/OPkPV83WHO7xj7G702u\n/vjvceaamNkCMxtpZifhXU03A8dLWjWVZ7qZ3WRmFfiA8gn4mKYQ6hXBSShbSTfKCfgX04MlKMLx\nktIz2k7CWxQy00UX4UFMetpwtyRfrlnk7+b5D7AicEprFLgZPsDHDWyUSZC0Cj4jpVWZWTU+++pA\nvEtnHvBAOo+k5XKOmY13T3RP5Vk2mWpbSJdKS2VaLtL3eHl80bRCFHX1VTP7EB9bcpRSU+Tl07vX\nrffAhj2CT3sfmjrfYvjv6EzgxSQt914ZPsAZkvuVJ893+Oy57oTQgJhKHMpNreZqM7u1voxtoBvw\nlKS78cGtJwLPm9lDyf6X8JaHWyT9M0k7jPxfSFXAgZL+gU9L/S45zy34ehqXSNoaH/C4NLALMNzM\nWisoq68b4E7gQuD+pA498IBwAj5rqrXdhU97PQl4LGeAL8A4Sc/gn9d0YEt8fZt/pvLsh88yOgr/\n/IrpUeCvwH8lXY+3Mh2Pz6KpsxhgHm3R/XI2/rm+IOkWYCX8d/VdmvcH6HB8yv0dkq7Ex70cjP8+\nnJAaeHubpO74LLDP8JaTU4BXk6AJfKzJf/EZSzPwacY/Bi5qRrlCJxLBSSg3hfylmW9NiJY+1yTf\n+U7B1xY5H1gcny30/RorZjZd0j74Gid/wv/zvRWfZvpYzvmuAjbFv1B/iTeNP2RmNZL2wr9gMoN/\nv8aDlHcKqF+h9c6bJ6nDvsAleJDyIb7OSD/qBictLQPASHxsRw9qz9LJuBwYgk+F7Y5/Tr8DLs5z\nzaZq6Ji8dTCzsZIOwO/vP/Av4UvxVp+rCrhGfdes73Ms9Njv95nZPZIOB87B7+EEPEg+CVg17xka\nuLaZzZI0GF+X5Wh83Z3xwKFmlr5nN+NjSU7Cg7bP8WDx/FSeS/EFCPfA7+eH+FTqywosV+ik1PKZ\nhSGEEMpNsjrs+2Y2tNHMIZSZko85kXSWfBnub+TLQ98nKd9grPqOHyRfGnt0McsZQgjlSL7kf5ec\ntD3xhdxGlaZUIbRMyVtO5M+iqMRXH1wMX+54I6C/meWb+pg+tifeNz0R6G1mxegjDyGEsiVpPXwm\nVCXetbIhPmV3KrBJM6f
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14607031ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzsnXecFFXywL8FIkgUQcCEoiKCCcGsiOjPfOaIAcF0iGdA\n71BRQfHuzHqggBFQVEyYsxhBQJRkAgOCqGTJQdLW74/qdnpmZ3ZnZmd3Z3fr+/n0p7tfv1Cve2a6\n5r2qeqKqOI7jOI7j5AvVylsAx3Ecx3GcKK6cOI7jOI6TV7hy4jiO4zhOXuHKieM4juM4eYUrJ47j\nOI7j5BWunDiO4ziOk1e4cuI4juM4Tl7hyonjOI7jOHmFKyeO4ziO4+QVrpw4GSEiBSLSJ8uys0Rk\nSOT8gqC+drmTMHtEZPtAnmvKW5aqjIh8LCJf5bjOuM9ePpPke9Ix+FwemsM2sv4eO05Z4MpJFSSi\nFBSIyEEp8vwaXH8t4ZIGWzYUJCmb0/UTRKSziFyVyzqd4gk+KwNyVF1prKmRVp2R70WBiGwUkd9F\n5F0R6VgKMqUimawZ3xMROVZE+hbRRpmvXSIifRPuceL9blLWMjn5ySblLYBTrqwBzgHGRhODH+Jt\ngD+TlNkM2JBle60wBaU0OQfYDehfyu04lZf3gCcBAVoAPYAPReQ4VX23rIVR1U9EZDNVXZdh0eMw\n2W9Ncq0k3+OSokB3YFWSa0vLWBYnT3HlpGrzFnCGiFypqlGl4RzgS6BxYoEsfiCjZddnW7aiIiK1\nVXV1ecvhZMQPqvpMeCIirwBfAVcDSZUTERFgU1VdWxoCZfm9kxzXl0tGquriTAqISE1gnSZZrTYX\n3zP/ruYXPq1TdVFgBNAIODJMFJEawOnAMyT5cUucqxaRW4K0nURkmIgsEZGlIjJERGollE01719H\nRB4WkUUiskxEnhCRzRPKnigibwTD7H+KyE8icpOIVIvk+Qg4HghtRwpE5OfI9ZqBvN+LyBoRmSMi\nI0WkRZJ+XhK08aeITBCRfYq7oZHpskNFZJCIzAd+Da4NE5GZScrcIiIFCWkFIjJARE4Ska8DGb4R\nkaOLab+JiKwXkZuTXNslqLdHcL5JMMT+Q3AvFonIaBE5orh+Zks6zzAhfzsR+UxEVovIzyLy9yR5\nNhWRW0Xkx6DO2SJyp4hsmiu5VfUbYBE2ihK2Gz6jc0TkG2yU8ejgmojI1cEzWyMi80TkocTPdJD3\nJrEp1FUi8oGItEmSJ6nNiYjsLyJvichiEVkpIlNF5Irg2lBs1CQ6VbUxQf4+CfXtLSJvB9/BFSIy\nSkT2T8gTfsYPEpH7RGRB0PZLItIo45ubgkifzxKRf4vIb9hISz0R6Zrqe5ZFP5LW4ZQ/PnJStZkF\njAc6E/tHeBxQH3gWSMd2I/wX8zzwM3A90A64GJgP3JAkbxQBHgSWAH2xqZ8eQHOgUyRfV2AFcC+w\nEjgc6AfUA64L8vwbaIBNSV0d1L0SIHgBvhnUOQL4X1D2SGB3IKo4nAvUBR4KZL4OGCkiO6rqRopn\nELAAG06vHel7KluCZOkdgFODulYAVwIvikhzVV2SrFFVXSAinwBnArclXD4bG8Z/Pji/FXtWjwBf\nYM98H+zZfVB8F7OiK8U/w5AtsOf1PKYonwkMFpG1qjoM/hqteB04CHgYmA7sAfQEWmL3r8SISEOg\nIfBjwqUjArkexJSXWUH6I0AXYAg2vdgCuAJoKyIHh58hEbkNuBF4A3gbu/fvATWSiBH3GRGRI7G+\nz8E+y/OA1sDfgAew+7E18H/Y5znlKEpQXxvgU2AZcAf2Wfk78LGIHKqqXyQUeQBYDNwC7IDd8wex\n35J0aBQ8vygbVHVZQtrNwFrgbqAmsI7YvYh+z+oE/dgtw34UqsPJE1TVtyq2ARcAG7Efwx7YPG/N\n4NpzwKjgeCbwWkLZAqBP5LxvkPZIQr6RwIKEtJnAkAQ5CoDPgeqR9H8G8v0tklYzST8GYy+7GpG0\n14Gfk+TtFrR1ZRH3ZfsgzwKgfiT9hECe49K4rwXAx4AkXBuaQq6+wMYk93gNsEMkbY8gvUcxMlwS\nyNomIf0b4P3I+eTEZ1vCz1QBMKCYPOk+w4+CPlwVSasBTALmhp8V4DxgPXBgQp2XBuUPSPXZK6Yf\nj2Ajio2B/YBRSeQpCNpulVD+kODaWQnpRwbpZwfnjbHRllcT8v07yBf9nnQM2j80OK+G/RGYAdQr\noi8PJH62EuSPfo9fDj5z20fSmmEv+Y+SfMbfSajvXkxxSClP5PNekGL7LqHPBZhCuGkG37NM+1Go\nDt/yY/NpHed57N/930SkLvbP6+kM61Dsn1qU0di/o7pplH9E40ckBhMoA381EJnLF5G6wRDymED2\nXdNo41RgIfbvrjieVdXlkfPR2D/PHdMoq8CjGvwCloD3VXXWX5Wqfg0sT0OGl7B7d1aYEPybbION\nhoUsBXYTkZ1LKGfaZPgMN2BKQlh2PfYZawK0D5JPB6YBP4hIo3DDlBshfuQtEy7CPisLsJHFA4F7\nVTXRyPpjVf0+Ie107N5+kCDTZGy0KJTpSEzheiCh/P/SkG9vbLTif6q6Ir0upSYYVTwSeFlVfwnT\nVXUeNmp1SML3WIk8m4DRQHVMwS8OBU7BRnWiW7ckeYdpcvuYQt+zLPuRi++qUwr4tE4VR1UXicgo\nzAi2Dvav7MUsqpqdcB5OPTQkmFpJJQLwU4JMq0RkLvYDDPw17Pwf7Me9fkL5BmnItxPwvcYb/qYi\nbu5ZVZcGI9AN0ygLseH9kpBs/ntJcTKo6h8i8gE23RC6kZ6N/ct/OZK1D/AK9mL/BngHGB4oQaVC\nhs9wjqquSUj7AVM6dgAmYFM3u2KKRCKKKTLZ8CqmxCo2qvNtElkg+XNuCWyOKTZFydQ82Cd+9heJ\nSNJpuwg7BXV9W0y+dNkSUxB/SHJtGvabsF1wHJL4+Yx+39NhtKZnEDsrg2vZ9KOo+p1yxJUTB+xf\nxaPAVsDbWf4bS2WLUeRcdzqISANsHnkpcBM2pP0n9g/6DnJv2F3SviR7kaX6d1a9FGR4FhgiInuq\n6lfAGcAH0ZeBqo4WkZ2Ak4CjsNGCniLyd1XNebCyUnqG1YCvMXuHZPclWwPH31T1wzTyJXvO1TBb\nq3NSyJRMkaqIlNr3PYFk9zida7mo3ylHXDlxwP5RPwzsT2Q6oIwQ7N/mJ38liNTBFKU3g6TDsH9k\nJ6nqZ5F8OyWpL5USMAPYT0Sqa3pGrblmCfaPOpEdSqGtV7DneVZgdLgLNmIRh6ouBZ4AnhCR2tjQ\n/C2YIWeuOYz0nyHA1mKxPaIvj1bY8w2Nl2cAe6rqR6Ugb7bMwAxlx2rRbsXhtENLIv/eRaQxxY8+\nzMC+N7sDRSlR6U5XLARWY/c3kdaYbUZF8GSpLP1wcFdiB5tGwYIi3YIZlJY1l4pIVFHugY0ovBWc\nb8R+jKNuw5sG+RJZRfJpnpHYsO8/ciFwFswAGojI7mGCiGwFnJzrhtQ8Ht7FpnbOxrwdXo3mEZEt\nEsqsxqYYakby1BeRViISnYLJlkyeIdgfp+6RvDUwr4uFmGEsmL3UtiJySWJhEakVKFxlzfOY7IVC\nw4tI9WAECczIdgPmxROlZxptTMIUtKsj9SVjVdBukc8vmOp8DzhJRMLpJkSkKeZ9M1pVi5qazQsq\nSz8cw0dOqi5xw6+qOry8BAE2xQwIn8dsCC7DfkjeCK6PxUYenpRYiPTzSP7PcCJwpojci7nIrgzq\neRJz77wviHkwGnMXPgIYqKq5UspSDWs/C9wJvBL0oQ728v0e85rKNc8BT2Ev/3cTDHwBvhORj7H7\ntRjYFzPmjIagPwXzMuqK3b/i2EdEbkyS/hGZPUMwr5xeIrIDZkNwNrAncElk5Gs4MRfjTsBnmFLb\nGpvKOoqYIlMmqOqnIvIwcL2ItMVeluux0avTMZfwlwLbknuCfG9givjewDEkn/r563OlqioilwGv\nAVPEYprMxb47bVT12CD
      "text/plain": [
       "<matplotlib.figure.Figure at 0x146072a3ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred_basic_model_bn = train_and_evaluate(reader_train, reader_test, max_epochs=5, model_func=create_basic_model_with_batch_normalization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's implement an inspired VGG style network, using layer API, here the architecture:\n",
    "\n",
    "| VGG9          |\n",
    "| ------------- |\n",
    "| conv3-64      |\n",
    "| conv3-64      |\n",
    "| max3          |\n",
    "|               |\n",
    "| conv3-96      |\n",
    "| conv3-96      |\n",
    "| max3          |\n",
    "|               |\n",
    "| conv3-128     |\n",
    "| conv3-128     |\n",
    "| max3          |\n",
    "|               |\n",
    "| FC-1024       |\n",
    "| FC-1024       |\n",
    "|               |\n",
    "| FC-10         |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_vgg9_model(input, out_dims):\n",
    "    with default_options(activation=relu):\n",
    "        model = Sequential([\n",
    "            For(range(3), lambda i: [\n",
    "                Convolution((3,3), [64,96,128][i], init=glorot_uniform(), pad=True),\n",
    "                Convolution((3,3), [64,96,128][i], init=glorot_uniform(), pad=True),\n",
    "                MaxPooling((3,3), strides=(2,2))\n",
    "            ]),\n",
    "            For(range(2), lambda : [\n",
    "                Dense(1024, init=glorot_uniform())\n",
    "            ]),\n",
    "            Dense(out_dims, init=glorot_uniform(), activation=None)\n",
    "        ])\n",
    "        \n",
    "    return model(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 2675978 parameters in 18 parameter tensors.\n",
      "\n",
      "Finished Epoch[1 of 300]: [Training] loss = 2.253115 * 50000, metric = 83.6% * 50000 46.007s (1086.8 samples per second);\n",
      "Finished Epoch[2 of 300]: [Training] loss = 1.931100 * 50000, metric = 71.8% * 50000 46.236s (1081.4 samples per second);\n",
      "Finished Epoch[3 of 300]: [Training] loss = 1.706618 * 50000, metric = 63.3% * 50000 46.271s (1080.6 samples per second);\n",
      "Finished Epoch[4 of 300]: [Training] loss = 1.576171 * 50000, metric = 58.1% * 50000 46.348s (1078.8 samples per second);\n",
      "Finished Epoch[5 of 300]: [Training] loss = 1.473403 * 50000, metric = 53.7% * 50000 46.386s (1077.9 samples per second);\n",
      "\n",
      "Final Results: Minibatch[1-626]: errs = 51.2% * 10000\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYVOX5//H3DSgoCigWMMGCiKJEFExiQ2NJbLHla9RF\njWKMBY2GJGr8GWtiibHE2LCLbdWY2A3Ye19QJCAqoFhQVKoUBfb+/XGfcc4Os7vDtpmd/byua66Z\nc85zznmeOQt771PN3REREREpFe2KnQERERGRNAUnIiIiUlIUnIiIiEhJUXAiIiIiJUXBiYiIiJQU\nBSciIiJSUhSciIiISElRcCIiIiIlRcGJiIiIlBQFJ1IWzKzazM5s4LkfmNlNqe3Dk+sNbLocNpyZ\nrZfk5/fFzovUzsxeMbNHG3juXWY2sanzVOC9G5xvkeai4ERKRiooqDazbWtJ81Fy/MGcQ568GqI6\nz7lNuq6DmVWY2UlNeU2pnZndnPpZqut1U/1XK1hjfgad+DksBq1hIiWnQ7EzIJLHQmAI8FJ6p5nt\nCHwPWJTnnJWAJQ2838Y0/y+GIcBmwOXNfB8JI4DHU9sbAOcC1wHPp/ZPbsJ7Dqbhv+gPBawJ8yLS\nqik4kVL0KPBLMzvR3dNBwxDgDWCN3BPc/duG3szdFzf03NbKzFZ29wXFzkdzcfdXgVcz22Y2CPgL\n8LK731nINcysk7vnC4Rru2dDg2PcfWlDzxUpR2rWkVLjQCXQHfhpZqeZrQAcANxJnr8wc/ucmNnZ\nyb4NzewWM5tlZrPN7CYz65Rz7ge1VO93NrNrzexLM5tjZiPNrFvOufuY2cNm9omZLTKz983sz2bW\nLpXmaWAvINN3pNrMpqSOd0zyO8nMFprZp2b2bzPbIE85f5PcY5GZvWZmW9X3haaay3Yws6vN7HPg\no+TYLWY2Nc85Z5tZdc6+ajP7p5nta2ZvJ3kYb2a71XP/tcxssZmdkedY3+S6w5LtDmZ2lpm9m3wX\nX5rZ82a2S33lbAwz+8zM7jGzvcysyswWAb9Kjv3GzJ4ys8+TPL1tZkfmuUaNvhtmtltStn2S7/MT\nM1tgZqPNbL2cc2v0OTGzjTPfS/KanNz7JTMbkOfeQ8xsYpLmzaQcDe7HYmZrJz8bM5JrjjWzijzp\nfmVmY8xsXvLv6y0zOy51fEUz+6uZvZdc5wsze9bMdmhIvqTtUM2JlKIPgFeACmB0sm9PoAtwF1BI\n341M9fo9wBTgT8BA4Cjgc+C0PGnTDLgSmAWcRTT9DAPWBXZKpTsCmAdcAnwN7Ew0H6wKnJqk+SvQ\nlWiS+l1y7a8BkiDmkeSalcA/knN/CvQH0oHDIcAqRJOFJ9f/t5n1LvAv76uBGcA5wMqpsucrf237\nBwO/SK41DzgRuNfM1nX3Wflu6u4zzOxZ4ECi9iLtYKI57p5k+xziWV0HvE48862IZ/dk/UVsMAc2\nB0YSZRsB/C85NizJy31E899+wA1m5u5+c8418jkL+Aa4kAi6TwFuoebPUW3f96+BTsBVQHvimd9r\nZn3d3QHM7BfA7USt4qlEzeJtwKd15KlWZtYZeIH4ef0n8DFwEHCHma3i7tcn6fZOyvFf4Frij93N\ngG2Aa5LLXUD8e70GGEv8O/gRsAXw3PLmTdoQd9dLr5J4AYcDS4lfRMOA2UDH5NjdwBPJ56nAgznn\nVgNnprbPSvZdl5Pu38CMnH1TgZty8lFNNAu0T+3/Y5K/n6f2dcxTjmuIX9wrpPY9BEzJk3Zocq8T\n6/he1kvSzAC6pPbvneRnzwK+12rgGcByjt1cS77OApbm+Y4XAuun9v0g2T+snjz8Jsnrpjn7xwOP\np7bH5j7bJvrZGpTk81e1HJ+e5G/7PMfyPeOngLdz9r0MPJra3i2555icn6OTk3v1Tu2rBCaktjP9\noD4BOqf2/zI5d+fUvknAe+l8EsFtdfqadXw3ufk+NbnHfql9HYjg5yugU+rn/LN6rj0RuKepn6de\n5f9Ss46UqnuIv+5/bmarAD8H7ljOazjxF13a80D35Jr1uc5r1khcQxIMfHcD928yn81sFTPrTvzV\nuTKwSQH3+AXwBVFLU5+73H1uavt5ohamdwHnOnC9uzd2ZMbj7v7Bdxd1fxuYW0Ae/kN8dwdldpjZ\nZsCmRG1YxmxgMzPr08h8NsREd38hd2fOM+5qZmsQf/X3M7MVC7juDTk/R5kOuYU8tzvcfX7Oud89\n86TpbyPg5nQ+3f1xImBpiD2AD939/tT1lgBXAN2AzEi62UBXM9u5jmvNBjbP10QpUhcFJ1KS3P1L\n4AmiE+wviJ/VextwqWk525mmh9XqywLwfk6e5hN/Ya+f2Wdmm5rZfWY2m/gl/QVRpQ5RhV2fDYFJ\nXrPjb20+ysnP7ORjfWXJ+KDAdAXnITGrvjy4+1dEs8yBqd0HA4uJ5pKMM4lfgO+a2Tgzu8jMftC4\nLBdsmb43EKPEzOxpM5tPlHVGkk8jmp3qk/udzUrOLeS55TuX1LmZviv5Rh29n2dfIdYD3s2zfyKR\n78w9rwA+BB43sw/N7Hoz2zXnnNOBtYHJSV+YC8xs0wbmS9oQBSdSyu4kaimOBf7r7vMacI3a+mI0\netimmXUl/oL+AfBnonZnV7J9TZr631djy7Iwz77aalLaN0Me7gL6mtnmyfYvgSfdfeZ3mXF/ngjY\nhgJvE30uxuTrgNoMlvl+zGwT4DGgM9F3Yk/iGWdqugp5xo35zprt57ex3P1T4md/f6Lf1K7AY2Z2\nTSrNU8Tz/DUR3BwDvGlmh7R8jqU1UXAipSzTAfHHRKDSkoyoLs/uiI6CPcnWQPyE+Av2cHe/0t0f\nTf4zns2yagsCJgMbm1ltwUBzm0XUVORavxnudT9RU3JQMuKkL9HXogZ3n+3uI939EKAXMA44uxny\nU4h9if4We7r7De4+KnnGpTL8/MPkPV8zWEObxj4knk2ufsTPceaeuPtid3/Q3YcRTU23AEeb2Tqp\nNDPd/WZ3ryA6lE8i+jSJ1ErBiZSspBnlWOIX00NFyMLRZpYe0TaMqFHIDBddSgQx6WHDKybpcs0n\nfzPPv4E1gROaIsMNMJnoN9A/s8PMehIjUpqUu88hRl8dSDTpfAM8kE5jZqvnnLOAaJ7omErTJRlq\nW0iTSmNlai7Sz7g7MWlaIZp19lV3n0r0LTnCUkPkLYZ3b1TriXV7lBj2vm/qeh2In9HZwIvJvtxn\n5UQHZ0ieV540XxOj5zoiUgcNJZZSU6O62t1vqy1hC1gReNLM7iE6tx4HPO/uDyfHXyJqHm41s38m\n+w4l/y+kKuBAM7uEGJb6dXKdW4n5NC41sx8THR5XAXYBrnL3pgrKamsGuAv4G3B/UobOREA4iRg1\n1dTuJoa9DgNG53TwBZhgZs8Q39dM4IfE/Db/TKXZnxhldATx/TWnUcD5wH/N7AailuloYhTNMpMB\n5tESzS+nE9/rC2Z2K7AW8bP6Pxr2B+hVxJD7O83sSqLfy8HEz8OxqY63t5tZR2IU2CdEzckJwKtJ\n0ATR1+S/xIilWcQw458DFzUgX9KGKDiRUlPIX5r55oRo7Lom+a53AjG3yDnACsRooe/mWHH3mWa2\nFzHHyV+I/3xvI4aZjs653tXAAOIX6u+IqvGH3b3azPYgfsFkOv9+RQQpbxdQvkLLnTdNUob9gEuJ\nIGUqMc9IX5YNThqbB4AHib4dnak5SifjcmAfYihsR+J7+n/AxXnuubzqOidvGdx9vJn9kni+lxC/\nhC8jan2uLuAetd2ztu+x0HO/O+bu95rZYcAZxDOcRATJw4B18l6hjnu7+3wzG0zMyzKUmHdnInCI\nu6ef2S1EX5JhRNA2nQgWz0mluYyYgHA34nlOJYZS/6PAfEkbZY0fWSgiIqUmmR32XXfft97EIiWm\n6H1OzOw0i2m451pMD32fmeXrjFXb+dtZTI09pjnzKSJSiiym/G+Xs293YiK3p4uTK5HGKXrNicVa\nFJXE7IMdiOmO+wP93D3
      "text/plain": [
       "<matplotlib.figure.Figure at 0x146072e6860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzsnXeYFGXSwH9FNCBgAswioiJGUIIBc8RwKieiZw4nZsyn\nntkzop8BFU8FMWA4PTHAyYF6IqjogiAqoAQTUZIkCbv1/VHddu/szO7M7Ozu7Gz9nqef7n777e56\npyfU1FtBVBXHcRzHcZx8oV5NC+A4juM4jhPHlRPHcRzHcfIKV04cx3Ecx8krXDlxHMdxHCevcOXE\ncRzHcZy8wpUTx3Ecx3HyCldOHMdxHMfJK1w5cRzHcRwnr3DlxHEcx3GcvMKVEycjRKRERG7O8tyZ\nIvJsbP/M4Hodcidh9ojINoE8V9a0LHUZEflQRCbm+Jql3nv5TJLPyQHB+7JbDu+R9efYcaoDV07q\nIDGloERE9knR56fg+FsJhzRYsqEkybk5rZ8gIr1E5PJcXtOpmOC98kiOLlcVNTXSumbsc1EiIsUi\n8ouIvCciB1SBTKlIJmvGr4mIHCUit5Rzj2qvXSIityS8xomvd4vqlsnJTxrUtABOjbISOBUYE28M\nvoi3AH5Pcs66wNos77cjpqBUJacC7YGHq/g+TuEyHBgECNAauAh4X0SOVtX3qlsYVf2fiKyrqqsz\nPPVoTPbbkhyrzOe4sihwIbA8ybHF1SyLk6e4clK3GQr8WUQuU9W40nAq8AWwSeIJWXxBxs9dk+25\ntRURWU9VV9S0HE5GTFXVl8IdEXkTmAhcASRVTkREgEaquqoqBMrycyc5vl4ueV1VF2Zygog0BlZr\nkmq1ufic+Wc1v/BpnbqLAoOBjYHDwkYRaQj0AF4iyZdb4ly1iNwatLURkYEiskhEFovIsyKyTsK5\nqeb91xeR/iLyq4gsEZHnRKR5wrnHicg7gZn9dxH5XkRuEpF6sT4fAN2B0HekRESmx443DuSdIiIr\nRWSWiLwuIq2TjPP84B6/i8hYEdmrohc0Nl3WTUQeF5G5wE/BsYEiMiPJObeKSElCW4mIPCIix4vI\nV4EMk0TkiAru30JE1ojI35Mc2yG47kXBfoPAxD41eC1+FZFRInJIRePMlnSeYUL/DiIyWkRWiMh0\nEflrkj6NROQ2EfkuuOaPInKviDTKldyqOgn4FbOihPcNn9GpIjIJszIeERwTEbkieGYrRWSOiDyZ\n+J4O+t4kNoW6XERGisjOSfok9TkRkc4iMlREForIMhGZICKXBscGYFaT+FRVcYL8Nydcb08RGRZ8\nBpeKyAgR6ZzQJ3yP7yMiD4rIvODeb4jIxhm/uCmIjbmniNwpIj9jlpYNROSsVJ+zLMaR9BpOzeOW\nk7rNTOBToBfRP8KjgabAy0A6vhvhv5hXgenA9UAH4DxgLvC3JH3jCPAYsAi4BZv6uQjYGjgo1u8s\nYCnQF1gGHAzcDmwAXBf0uRNohk1JXRFcexlA8AP4bnDNwcD/BeceBuwCxBWH04AmwJOBzNcBr4vI\ndqpaTMU8DszDzOnrxcaeypcgWfv+wInBtZYClwH/EpGtVXVRspuq6jwR+R9wMnBHwuFTMDP+q8H+\nbdizegr4HHvme2HPbmTFQ8yKs6j4GYZshD2vVzFF+WTgCRFZpaoD4Q9rxdvAPkB/YDKwK9AHaIu9\nfpVGRDYENgS+Szh0SCDXY5jyMjNofwo4A3gWm15sDVwK7CEi+4bvIRG5A7gReAcYhr32w4GGScQo\n9R4RkcOwsc/C3stzgHbAMcCj2OuxOXAo9n5OaUUJrrcz8BGwBLgHe6/8FfhQRLqp6ucJpzwKLARu\nBbbFXvPHsO+SdNg4eH5x1qrqkoS2vwOrgPuBxsBqotci/jlbPxhH+wzHUeYaTp6gqr7UsQU4EyjG\nvgwvwuZ5GwfHXgFGBNszgLcSzi0Bbo7t3xK0PZXQ73VgXkLbDODZBDlKgM+A+rH2qwP5jom1NU4y\njiewH7uGsba3gelJ+p4d3Ouycl6XbYI+84CmsfZjA3mOTuN1LQE+BCTh2IAUct0CFCd5jVcC28ba\ndg3aL6pAhvMDWXdOaJ8E/De2Pz7x2VbyPVUCPFJBn3Sf4QfBGC6PtTUExgGzw/cK8BdgDdA14ZoX\nBOd3SfXeq2AcT2EWxU2ATsCIJPKUBPfeMeH8/YJjPRPaDwvaTwn2N8GsLUMS+t0Z9It/Tg4I7t8t\n2K+H/RGYBmxQzlgeTXxvJcgf/xz/O3jPbRNra4X9yH+Q5D3+n4Tr9cUUh5TyxN7vJSmWbxLGXIIp\nhI0y+JxlOo4y1/AlPxaf1nFexf7dHyMiTbB/Xi9meA3F/qnFGYX9O2qSxvlPaWmLxBMEysAfN4jN\n5YtIk8CE/HEg+05p3ONEYD72764iXlbV32L7o7B/ntulca4C/9TgG7AS/FdVZ/5xUdWvgN/SkOEN\n7LXrGTYE/yZ3xqxhIYuB9iKyfSXlTJsMn+FaTEkIz12DvcdaAB2D5h7At8BUEdk4XDDlRihtecuE\nc7H3yjzMstgV6KuqiU7WH6rqlIS2HthrOzJBpvGYtSiU6TBM4Xo04fz/S0O+PTFrxf+p6tL0hpSa\nwKp4GPBvVf0hbFfVOZjVar+Ez7ESezYBo4D6mIJfEQqcgFl14svZSfoO1OT+MWU+Z1mOIxefVacK\n8GmdOo6q/ioiIzAn2PWxf2X/yuJSPybsh1MPGxJMraQSAfg+QablIjIb+wIG/jA734V9uTdNOL9Z\nGvK1AaZoacffVJSae1bVxYEFesM0zoXIvF8Zks1/L6pIBlVdICIjsemGMIz0FOxf/r9jXW8G3sR+\n2CcB/wGeD5SgKiHDZzhLVVcmtE3FlI5tgbHY1M1OmCKRiGKKTDYMwZRYxaw6XyeRBZI/57ZAc0yx\nKU+mrYN14nv/VxFJOm0Xo01wra8r6Jcum2IK4tQkx77FvhO2CrZDEt+f8c97OozS9BxiZ2ZwLJtx\nlHd9pwZx5cQB+1fxT2AzYFiW/8ZS+WKUO9edDiLSDJtHXgzchJm0f8f+Qd9D7h27KzuWZD9kqf6d\n1a8CGV4GnhWR3VR1IvBnYGT8x0BVR4lIG+B44HDMWtBHRP6qqjlPVlZFz7Ae8BXm75DsdcnWwfFn\nVX0/jX7JnnM9zNfq1BQyJVOkaiNV9nlPINlrnM6xXFzfqUFcOXHA/lH3BzoTmw6oJgT7t/m/PxpE\n1scUpXeDpgOxf2THq+roWL82Sa6XSgmYBnQSkfqanlNrrlmE/aNOZNsquNeb2PPsGTgd7oBZLEqh\nqouB54DnRGQ9zDR/K+bImWsOJP1nCLC5WG6P+I/HjtjzDZ2XpwG7qeoHVSBvtkzDHGXHaPlhxeG0\nQ1ti/95FZBMqtj5Mwz43uwDlKVHpTlfMB1Zgr28i7TDfjNoQyVIo43DwUGIHm0bBkiLdijmUVjcX\niEhcUb4IsygMDfaLsS/jeNhwo6BfIstJPs3zOmb2vSQXAmfBNKCZiOwSNojIZsCfcn0jtYiH97Cp\nnVOwaIch8T4islHCOSuwKYbGsT5NRWRHEYlPwWRLJs8Q7I/ThbG+DbGoi/mYYyyYv9SWInJ+4ski\nsk6gcFU3r2Kyl0kNLyL1AwsSmJPtWiyKJ06fNO4xDlPQrohdLxnLg/uW+/yCqc7hwPEiEk43ISIt\nseibUapa3tRsXlAo43AMt5zUXUqZX1X1+ZoSBGiEORC+ivkQ9Ma+SN4Jjo/BLA+DJEqR/heS/zMs\nAk4Wkb5YiOyy4DqDsPDOB4OcB6OwcOFDgH6qmiulLJVZ+2XgXuDNYAzrYz++U7CoqVzzCvAC9uP/\nXoKDL8A3IvIh9notBPbGnDnjKehPwKKMzsJev4rYS0RuTNL+AZk9Q7ConGtFZFvMh+AUYDfg/Jjl\n63miEOODgNGYUtsOm8o6nEiRqRZU9SMR6Q9cLyJ7YD+WazDrVQ8sJPyNwLfkgaDfO5gividwJMmn\nfv54X6mqikhv4C3gS7G
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1460735c908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred_vgg = train_and_evaluate(reader_train, reader_test, max_epochs=5, model_func=create_vgg9_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Residual Network (ResNet)\n",
    "\n",
    "One of the main problem of a Deep Neural Network is how to propagate the error all the way to the first layer. For a deep network, the gradient keep getting smaller until it has no effect on the network weights. [ResNet](https://arxiv.org/abs/1512.03385) was designed to overcome such problem, by defining a block with identity path, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/201/ResNetBlock2.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 7\n",
    "Image(url=\"https://cntk.ai/jup/201/ResNetBlock2.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of the above block is 2 folds:\n",
    "\n",
    "* During back propagation the gradient have a path that doesn't affect its magnitude.\n",
    "* The network need to learn residual mapping (delta to x).\n",
    "\n",
    "So let's implements ResNet blocks using CNTK:\n",
    "\n",
    "            ResNetNode                   ResNetNodeInc\n",
    "                |                              |\n",
    "         +------+------+             +---------+----------+\n",
    "         |             |             |                    |\n",
    "         V             |             V                    V\n",
    "    +----------+       |      +--------------+   +----------------+\n",
    "    | Conv, BN |       |      | Conv x 2, BN |   | SubSample, BN  |\n",
    "    +----------+       |      +--------------+   +----------------+\n",
    "         |             |             |                    |\n",
    "         V             |             V                    |\n",
    "     +-------+         |         +-------+                |\n",
    "     | ReLU  |         |         | ReLU  |                |\n",
    "     +-------+         |         +-------+                |\n",
    "         |             |             |                    |\n",
    "         V             |             V                    |\n",
    "    +----------+       |        +----------+              |\n",
    "    | Conv, BN |       |        | Conv, BN |              |\n",
    "    +----------+       |        +----------+              |\n",
    "         |             |             |                    |\n",
    "         |    +---+    |             |       +---+        |\n",
    "         +--->| + |<---+             +------>+ + +<-------+\n",
    "              +---+                          +---+\n",
    "                |                              |\n",
    "                V                              V\n",
    "            +-------+                      +-------+\n",
    "            | ReLU  |                      | ReLU  |\n",
    "            +-------+                      +-------+\n",
    "                |                              |\n",
    "                V                              V\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from cntk.ops import combine, times, element_times, AVG_POOLING\n",
    "\n",
    "def convolution_bn(input, filter_size, num_filters, strides=(1,1), init=he_normal(), activation=relu):\n",
    "    if activation is None:\n",
    "        activation = lambda x: x\n",
    "        \n",
    "    r = Convolution(filter_size, num_filters, strides=strides, init=init, activation=None, pad=True, bias=False)(input)\n",
    "    r = BatchNormalization(map_rank=1)(r)\n",
    "    r = activation(r)\n",
    "    \n",
    "    return r\n",
    "\n",
    "def resnet_basic(input, num_filters):\n",
    "    c1 = convolution_bn(input, (3,3), num_filters)\n",
    "    c2 = convolution_bn(c1, (3,3), num_filters, activation=None)\n",
    "    p  = c2 + input\n",
    "    return relu(p)\n",
    "\n",
    "def resnet_basic_inc(input, num_filters):\n",
    "    c1 = convolution_bn(input, (3,3), num_filters, strides=(2,2))\n",
    "    c2 = convolution_bn(c1, (3,3), num_filters, activation=None)\n",
    "\n",
    "    s = convolution_bn(input, (1,1), num_filters, strides=(2,2), activation=None)\n",
    "    \n",
    "    p = c2 + s\n",
    "    return relu(p)\n",
    "\n",
    "def resnet_basic_stack(input, num_filters, num_stack):\n",
    "    assert (num_stack > 0)\n",
    "    \n",
    "    r = input\n",
    "    for _ in range(num_stack):\n",
    "        r = resnet_basic(r, num_filters)\n",
    "    return r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write the full model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_resnet_model(input, out_dims):\n",
    "    conv = convolution_bn(input, (3,3), 16)\n",
    "    r1_1 = resnet_basic_stack(conv, 16, 3)\n",
    "\n",
    "    r2_1 = resnet_basic_inc(r1_1, 32)\n",
    "    r2_2 = resnet_basic_stack(r2_1, 32, 2)\n",
    "\n",
    "    r3_1 = resnet_basic_inc(r2_2, 64)\n",
    "    r3_2 = resnet_basic_stack(r3_1, 64, 2)\n",
    "\n",
    "    # Global average pooling\n",
    "    pool = AveragePooling(filter_shape=(8,8), strides=(1,1))(r3_2)    \n",
    "    net = Dense(out_dims, init=he_normal(), activation=None)(pool)\n",
    "    \n",
    "    return net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 272474 parameters in 65 parameter tensors.\n",
      "\n",
      "Finished Epoch[1 of 300]: [Training] loss = 1.859668 * 50000, metric = 69.3% * 50000 47.499s (1052.7 samples per second);\n",
      "Finished Epoch[2 of 300]: [Training] loss = 1.583096 * 50000, metric = 58.7% * 50000 48.541s (1030.0 samples per second);\n",
      "Finished Epoch[3 of 300]: [Training] loss = 1.453993 * 50000, metric = 53.4% * 50000 48.982s (1020.8 samples per second);\n",
      "Finished Epoch[4 of 300]: [Training] loss = 1.347815 * 50000, metric = 49.2% * 50000 48.704s (1026.6 samples per second);\n",
      "Finished Epoch[5 of 300]: [Training] loss = 1.269185 * 50000, metric = 45.8% * 50000 48.155s (1038.3 samples per second);\n",
      "\n",
      "Final Results: Minibatch[1-626]: errs = 44.6% * 10000\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYVOXZx/HvDxAREBQL9i6KYgNrLNhbIqhR49pLNIpG\nxZbXGFsSgzGxK3YBo25ibNhFo8YWG2BHTUBEQEWlIyiy9/vHfcY5O8zuzvazu/fnuuaaOc9pzzNn\nYe99qsyMEEIIIYSsaNfcGQghhBBCSIvgJIQQQgiZEsFJCCGEEDIlgpMQQgghZEoEJyGEEELIlAhO\nQgghhJApEZyEEEIIIVMiOAkhhBBCpkRwEkIIIYRMieAktAqSKiRdWMdzJ0q6I7V9dHK9vg2Xw7qT\ntGaSnzObOy+hapJelfR4Hc/9u6RxDZ2nEu9d53yH0FgiOAmZkQoKKiT9pIpjPkv2P1ywy5JXXVQU\nObdB13WQVCbp9Ia8ZqiapGGpn6XqXnfUfLWS1edn0PCfw+YQa5iEzOnQ3BkIoYj5wGHAK+lESf2B\nVYEFRc5ZCvihjvfbgMb/xXAYsDFwTSPfJ7ibgKdT22sDvwduAV5MpY9vwHvuSN1/0R8BqAHzEkKL\nFsFJyKLHgYMlnWZm6aDhMOBNYPnCE8zs+7rezMwW1vXclkpSZzP7trnz0VjM7DXgtdy2pH7AH4D/\nmNk9pVxDUiczKxYIV3XPugbHmNmiup4bQmsUzTohawwoB5YD9sglSloCOAi4hyJ/YRb2OZF0cZK2\nrqThkmZIminpDkmdCs6dWEX1fhdJN0v6WtIsSSMkLVNw7gBJj0qaImmBpP9J+p2kdqljngN+CuT6\njlRImpDav2SS348kzZc0VdL9ktYuUs4TknsskPS6pC1r+kJTzWU7SRoq6Uvgs2TfcEmfFDnnYkkV\nBWkVkq6VNFDSu0ke3pO0Vw33X1HSQkkXFNnXK7nuoGS7g6SLJH2cfBdfS3pR0m41lbM+JH0h6V5J\nP5U0WtIC4Khk3wmSnpX0ZZKndyUdV+QalfpuSNorKduA5PucIulbSU9JWrPg3Ep9TiRtkPtektf4\n5N6vSNqsyL0PkzQuOeatpBx17sciqWfyszEtueZYSWVFjjtK0hhJc5J/X29LOjm1v6OkP0r6b3Kd\nryT9W9JOdclXaDui5iRk0UTgVaAMeCpJ2xfoBvwdKKXvRq56/V5gAvB/QF/gl8CXwHlFjk0TcD0w\nA7gIb/oZBKwB7JI67hhgDnAFMBfYFW8+WBr4TXLMH4HueJPUGcm15wIkQcxjyTXLgauTc/cA+gDp\nwOFwoCveZGHJ9e+XtE6Jf3kPBaYBlwCdU2UvVv6q0ncEDkyuNQc4DbhP0hpmNqPYTc1smqR/A4fg\ntRdph+LNcfcm25fgz+oW4A38mW+JP7t/1VzEOjNgU2AEXrabgPeTfYOSvDyIN//tD9wmycxsWME1\nirkI+A64DA+6zwWGU/nnqKrv+3igE3AD0B5/5vdJ6mVmBiDpQOAuvFbxN3jN4t+AqdXkqUqSugAv\n4T+v1wKTgV8Ad0vqama3Jsftl5TjCeBm/I/djYHtgBuTyw3B/73eCIzF/x1sDWwOvFDbvIU2xMzi\nFa9MvICjgUX4L6JBwExgyWTfP4Bnks+fAA8XnFsBXJjavihJu6XguPuBaQVpnwB3FOSjAm8WaJ9K\nPzvJ389SaUsWKceN+C/uJVJpjwATihx7bHKv06r5XtZMjpkGdEul75fkZ98SvtcK4HlABfuGVZGv\ni4BFRb7j+cBaqbRNkvRBNeThhCSvGxWkvwc8ndoeW/hsG+hnq1+Sz6Oq2P95kr8diuwr9oyfBd4t\nSPsP8Hhqe6/knmMKfo7OSe61TiqtHPggtZ3rBzUF6JJKPzg5d9dU2kfAf9P5xIPbivQ1q/luCvP9\nm+Qe+6fSOuDBzzdAp9TP+Rc1XHsccG9DP894tf5XNOuErLoX/+v+Z5K6Aj8D7q7lNQz/iy7tRWC5\n5Jo1ucUq10jcSBIM/HgDs+9ynyV1lbQc/ldnZ2DDEu5xIPAVXktTk7+b2ezU9ot4Lcw6JZxrwK1m\nVt+RGU+b2cQfL2r2LjC7hDw8gH93v8glSNoY2AivDcuZCWwsab165rMuxpnZS4WJBc+4u6Tl8b/6\ne0vqWMJ1byv4Ocp1yC3lud1tZvMKzv3xmSdNf+sDw9L5NLOn8YClLvYBPjWzh1LX+wG4DlgGyI2k\nmwl0l7RrNdeaCWxarIkyhOpEcBIyycy+Bp7BO8EeiP+s3leHS00q2M41PSxbUxaA/xXkaR7+F/Za\nuTRJG0l6UNJM/Jf0V3iVOngVdk3WBT6yyh1/q/JZQX5mJh9rKkvOxBKPKzkPiRk15cHMvsGbZQ5J\nJR8KLMSbS3IuxH8BfizpHUmXS9qkflku2WJ9b8BHiUl6TtI8vKzTknwKb3aqSeF3NiM5t5TnVuxc\nUufm+q4UG3X0vyJppVgT+LhI+jg837l7Xgd8Cjwt6VNJt0raveCc84GewPikL8wQSRvVMV+hDYng\nJGTZPXgtxUnAE2Y2pw7XqKovRr2HbUrqjv8FvQnwO7x2Z3fyfU0a+t9Xfcsyv0haVTUp7RshD38H\neknaNNk+GPiXmU3/MTNmL+IB27HAu3ifizHFOqA2gsW+H0kbAqOALnjfiX3xZ5yr6SrlGdfnO2u0\nn9/6MrOp+M/+AXi/qd2BUZJuTB3zLP48j8eDm18Bb0k6vOlzHFqSCE5CluU6IG6DBypNSXh1eT7B\nOwquTL4GYmf8L9ijzex6M3s8+c94JourKggYD2wgqapgoLHNwGsqCq3VCPd6CK8p+UUy4qQX3tei\nEjObaWYjzOxwYHXgHeDiRshPKQbi/S32NbPbzOzJ5BlnZfj5p8l7sWawujaNfYo/m0K98Z/j3D0x\ns4Vm9rCZDcKbmoYDJ0paJXXMdDMbZmZleIfyj/A+TSFUKYKTkFlJM8pJ+C+mR5ohCydKSo9oG4TX\nKOSGiy7Cg5j0sOGOyXGF5lG8med+YAXg1IbIcB2Mx/sN9MklSFoZH5HSoMxsFj766hC8Sec7YGT6\nGEk9Cs75Fm+eWDJ1TLdkqG0pTSr1lau5SD/j5fBJ00rRqLOvmtkneN+SY5QaIi8f3r1+lSdW73F8\n2PvA1PU64D+jM4GXk7TCZ2V4B2dInleRY+bio+eWJIRqxFDikDWVqqvN7G9VHdgEOgL/knQv3rn1\nZOBFM3s02f8KXvNwp6Rrk7QjKP4LaTRwiKQr8GGpc5Pr3InPp3GlpG3wDo9dgd2AG8ysoYKyqpoB\n/g78GXgoKUMXPCD8CB811dD+gQ97HQQ8VdDBF+ADSc/j39d0YCt8fptrU8ccgI8yOgb//hrTk8Cf\ngCck3YbXMp2Ij6JZbDLAIpqi+eV8/Ht9SdKdwIr4z+r71O0P0BvwIff3SLoe7/dyKP7zcFKq4+1d\nkpbER4FNwWtOTgVeS4Im8L4mT+Ajlmbgw4x/Blxeh3yFNiSCk5A1pfylWWxOiPqua1Lseqfic4tc\nAiyBjxb6cY4VM5su6af4HCd/wP/z/Rs+zPSpgusNBTbDf6GegVeNP2pmFZL2wX/B5Dr/foMHKe+W\nUL5Sy130mKQM+wNX4kHKJ/g8I71YPDipbx4AHsb7dnSh8iidnGuAAfhQ2CXx7+m3wF+L3LO2qjun\naBnM7D1JB+PP9wr8l/BVeK3P0BLuUdU9q/oeSz33x31mdp+kI4EL8Gf4ER4kDwJWKXqFau5tZvMk\n7YjPy3IsPu/OOOBwM0s/s+F4X5JBeND2OR4sXpI65ip8AsK98Of5CT6U+uoS8xXaKNV/ZGEIIYSs\nSWaH/djMBtZ4cAgZ0+x9TiSdJ5+Ge7Z8eugHJRXrjFV4XkdJl8qnHl8gaYKkY5ogyyGEkBnyKf/b\nFaTtjU/k9lzz5CqE+slCs86O+Hj5N/H8DMGHo/U2s2JDH3P+iXckPBbv1LcyGQi2Qgihia2L9xkq\nx5tWNsaH7H4K3N6cGQu
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14606ffbd30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAADeCAYAAADmUqAlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzsnWeYFMXWgN8DCBhRVMSACCoCYgIEwxVzuCbMCqJivMaL\nqNesGK75AzOKegETGFEwopglShBUQFGCSEYQyWnP9+N0072zs7szs7O7s7vnfZ5+uru6qvrU9M72\nmaoTRFVxHMdxHMfJFaqVtwCO4ziO4zhxXDlxHMdxHCencOXEcRzHcZycwpUTx3Ecx3FyCldOHMdx\nHMfJKVw5cRzHcRwnp3DlxHEcx3GcnMKVE8dxHMdxcgpXThzHcRzHySlcOXHSQkTyROTODNtOF5He\nsfMLgv5aZk/CzBGRhoE815W3LFUZEflSRCZkuc98f3u5TJLvyaHB32W7LN4j4++x45QFrpxUQWJK\nQZ6IHFRInZnB9UEJlzTYMiEvSdus5k8QkQ4i0iWbfTrFE/ytPJGl7kojp0ZKfca+F3kisl5EZonI\nYBE5tBRkKoxksqb9mYjIP0WkWxH3KPPcJSLSLeEzTvy865W1TE5uUqO8BXDKlZVAR2BYvDD4R7wj\nsCpJm42BdRnebw9MQSlNOgJ7Ao+X8n2cyssnwEuAAI2AK4HPReR4VR1c1sKo6lcisrGqrkmz6fGY\n7HcnuVaS73FJUeByYHmSa3+VsSxOjuLKSdXmQ+BMEfm3qsaVho7AaGCbxAYZ/IOMt12baduKiohs\noqorylsOJy1+UdV+4YmIvAtMAK4FkionIiJATVVdXRoCZfi9kyz3l03eVtVF6TQQkVrAGk2SrTYb\n3zP/ruYWvqxTdVGgP7A1cHRYKCIbAWcA/Ujyzy1xrVpE7grKdhWRviKyWET+EpHeIlI7oW1h6/6b\nikgvEVkoIktE5EUR2TKh7cki8n4wzb5KRH4VkdtFpFqszhfACUBoO5InIlNj12sF8v4sIitFZLaI\nvC0ijZKM89LgHqtEZJSItC7uA40tl7UTkZ4iMg+YGVzrKyLTkrS5S0TyEsryROQJEWkvIj8EMvwo\nIscWc/96IrJWRO5Icq1J0O+VwXmNYIr9l+CzWCgi34jIkcWNM1NSeYYJ9VuKyFARWSEiU0XkX0nq\n1BSRu0VkStDn7yLykIjUzJbcqvojsBCbRQnvGz6jjiLyIzbLeGxwTUTk2uCZrRSRuSLybOLfdFD3\ndrEl1OUi8pmINE9SJ6nNiYi0FZEPRWSRiCwTkfEick1wrQ82axJfqlqfIP+dCf3tJyIfBd/BpSIy\nRETaJtQJ/8YPEpEeIjI/uPcAEdk67Q+3EGJjPltE/isif2AzLZuLSOfCvmcZjCNpH0754zMnVZvp\nwAigA9EvwuOBLYDXgFRsN8JfMW8AU4GbgZbAJcA84JYkdeMI8BSwGOiGLf1cCewMHB6r1xlYCnQH\nlgFHAPcAmwM3BXX+C9TBlqSuDfpeBhC8AD8I+uwPPBa0PRpoAcQVh3OBzYBnA5lvAt4Wkcaqup7i\n6QnMx6bTN4mNvTBbgmTlhwCnBX0tBf4NvCUiO6vq4mQ3VdX5IvIVcBZwb8Llc7Bp/DeC87uxZ/Uc\n8B32zFtjz+6z4oeYEZ0p/hmG1MWe1xuYonwW8IyIrFbVvrBhtuI94CCgFzAZ2AvoCuyOfX4lRkS2\nArYCpiRcOjKQ6ylMeZkelD8HnA/0xpYXGwHXAPuKyMHh35CI3AvcBrwPfIR99p8AGyURI9/fiIgc\njY19Nva3PBdoBpwIPIl9HjsAR2F/z4XOogT9NQe+BpYAD2J/K/8CvhSRdqr6XUKTJ4FFwF3ALthn\n/hT2vyQVtg6eX5x1qrokoewOYDXwCFALWEP0WcS/Z5sG49gzzXEU6MPJEVTVtyq2ARcA67F/hldi\n67y1gmuvA0OC42nAoIS2ecCdsfNuQdlzCfXeBuYnlE0DeifIkQeMBKrHym8I5DsxVlYryTiewV52\nG8XK3gOmJql7YXCvfxfxuTQM6swHtoiVnxTIc3wKn2se8CUgCdf6FCJXN2B9ks94JbBLrGyvoPzK\nYmS4NJC1eUL5j8CnsfNxic+2hH9TecATxdRJ9Rl+EYyhS6xsI2AsMCf8WwE6AWuBAxP6vCxof0Bh\nf3vFjOM5bEZxG6ANMCSJPHnBvfdIaP+P4NrZCeVHB+XnBOfbYLMtAxPq/TeoF/+eHBrcv11wXg37\nIfAbsHkRY3ky8W8rQf749/id4G+uYaysPvaS/yLJ3/jHCf11xxSHQuWJ/b3nFbJNTBhzHqYQ1kzj\ne5buOAr04VtubL6s47yB/bo/UUQ2w355vZpmH4r9UovzDfbraLMU2j+n+WckniFQBjbcILaWLyKb\nBVPI3wayN03hHqcBC7Bfd8Xxmqr+HTv/Bvvl2TiFtgo8r8F/wBLwqapO39Cp6g/A3ynIMAD77M4O\nC4Jfk82x2bCQv4A9RWS3EsqZMmk+w3WYkhC2XYv9jdUDWgXFZwCTgF9EZOtww5QbIf/MWzpcjP2t\nzMdmFg8EuqtqopH1l6r6c0LZGdhn+1mCTOOw2aJQpqMxhevJhPaPpSDffthsxWOqujS1IRVOMKt4\nNPCOqs4Iy1V1LjZr9Y+E77ESezYB3wDVMQW/OBQ4FZvViW8XJqnbV5PbxxT4nmU4jmx8V51SwJd1\nqjiqulBEhmBGsJtiv8reyqCr3xPOw6WHrQiWVgoTAfg1QablIjIH+wcMbJh2vg/7575FQvs6Kci3\nK/Cz5jf8LYx8a8+q+lcwA71VCm0hmt4vCcnWvxcXJ4Oq/ikin2HLDaEb6TnYr/x3YlXvBN7FXuw/\nAh8DLwdKUKmQ5jOcraorE8p+wZSOXYBR2NJNU0yRSEQxRSYTBmJKrGKzOj8lkQWSP+fdgS0xxaYo\nmXYO9ol/+wtFJOmyXYxdg75+KqZeqmyLKYi/JLk2Cfuf0CA4Dkn8+4x/31PhG03NIHZ6GtcyGUdR\n/TvliCsnDtiviueB7YGPMvw1VpgtRpFr3akgInWwdeS/gNuxKe1V2C/oB8m+YXdJx5LsRVbYr7Pq\npSDDa0BvEdlbVScAZwKfxV8GqvqNiOwKtAeOwWYLuorIv1Q168HKSukZVgN+wOwdkn0umRo4/qGq\nn6dQL9lzrobZWnUsRKZkilRFpNS+7wkk+4xTuZaN/p1yxJUTB+wXdS+gLbHlgDJCsF+bX20oENkU\nU5Q+CIoOw36RtVfVobF6uybprzAl4DegjYhU19SMWrPNYuwXdSK7lMK93sWe59mB0WETbMYiH6r6\nF/Ai8KKIbIJNzd+FGXJmm8NI/RkC7CAW2yP+8tgDe76h8fJvwN6q+kUpyJspv2GGssO0aLficNlh\nd2K/3kVkG4qfffgN+960AIpSolJdrlgArMA+30SaYbYZFcGTpbKMw8FdiR1sGQULinQXZlBa1lwm\nInFF+UpsRuHD4Hw99s847jZcM6iXyHKSL/O8jU37Xp0NgTPgN6COiLQIC0Rke+CUbN9IzeNhMLa0\ncw7m7TAwXkdE6ia0WYEtMdSK1dlCRPYQkfgSTKak8wzBfjhdHqu7EeZ1sQAzjAWzl9pJRC5NbCwi\ntQOFq6x5A5O9QGh4EakezCCBGdmuw7x44nRN4R5jMQXt2lh/yVge3LfI5xcsdX4CtBeRcLkJEdkO\n8775RlWLWprNCSrLOBzDZ06qLvmmX1X15fISBKiJGRC+gdkQXIH9I3k/uD4Mm3l4SaIQ6Z1I/stw\nDHCWiHTHXGSXBf28hLl39ghiHnyDuQsfCTytqtlSygqb1n4NeAh4NxjDptjL92fMayrbvA68gr38\nBycY+AJMFJEvsc9rEbA/ZswZD0F/KuZl1Bn7/IqjtYjclqT8C9J7hmBeOTeKyC6YDcE5wN7ApbGZ\nr5eJXIwPB4ZiSm0zbCnrGCJFpkxQ1a9FpBdws4jsi70s12KzV2dgLuEDAtuS/wvqvY8p4vsBx5F8\n6WfD35WqqohcAQwCvhe
      "text/plain": [
       "<matplotlib.figure.Figure at 0x146074da400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred_resnet = train_and_evaluate(reader_train, reader_test, max_epochs=5, model_func=create_resnet_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}