зеркало из https://github.com/microsoft/caffe.git
[examples] flickr fine-tuning notebook
This commit is contained in:
Yangqing Jia
Evan Shelhamer
Родитель
ccda10250d
Коммит
b8cc2974fc
|
|
@ -0,0 +1,951 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Finetune a Pretrained Network with Flickr Style Data\n",
|
||||
"\n",
|
||||
"In this example, we'll explore a common approach that is particularly useful in real-world applications: take a pre-trained Caffe network, and finetune the last few layers using your custom data.\n",
|
||||
"\n",
|
||||
"The upside of such approach is that, since pre-trained networks are trained on a large set of images, the intermediate layers capture the \"semantics\" of the general visual appearance. Think of it as a very powerful feature that you can treat as a black box. On top of that, only a few layers will be needed to obtain a very good performance of the data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First, we will need to prepare the data. This involves the following parts:\n",
|
||||
"(1) Get the ImageNet ilsvrc pretrained model with the provided shell scripts.\n",
|
||||
"(2) Download a subset of the overall Flickr style dataset for this demo.\n",
|
||||
"(3) Compile the downloaded Flickr dataset into a database that Caffe can then consume."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/jiayq/Research/caffe\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"os.chdir('..')\n",
|
||||
"import sys\n",
|
||||
"sys.path.insert(0, './python')\n",
|
||||
"print os.getcwd()\n",
|
||||
"\n",
|
||||
"import caffe\n",
|
||||
"import numpy as np\n",
|
||||
"from pylab import *\n",
|
||||
"%matplotlib inline"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Downloading...\n",
|
||||
"--2015-03-17 10:51:07-- http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz\n",
|
||||
"Resolving dl.caffe.berkeleyvision.org (dl.caffe.berkeleyvision.org)... 169.229.222.251\n",
|
||||
"Connecting to dl.caffe.berkeleyvision.org (dl.caffe.berkeleyvision.org)|169.229.222.251|:80... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 17858008 (17M) [application/octet-stream]\n",
|
||||
"Saving to: ‘caffe_ilsvrc12.tar.gz’\n",
|
||||
"\n",
|
||||
"100%[======================================>] 17,858,008 287KB/s in 55s \n",
|
||||
"\n",
|
||||
"2015-03-17 10:52:02 (318 KB/s) - ‘caffe_ilsvrc12.tar.gz’ saved [17858008/17858008]\n",
|
||||
"\n",
|
||||
"Unzipping...\n",
|
||||
"Done.\n",
|
||||
"Model already exists.\n",
|
||||
"Downloading 2000 images with 3 workers...\n",
|
||||
"Writing train/val for 1903 successfully downloaded images.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# This downloads the ilsvrc auxiliary data (mean file, etc),\n",
|
||||
"# and a subset of 2000 images for the style recognition task.\n",
|
||||
"\n",
|
||||
"# You won't need to run this - we should have already created it for you.\n",
|
||||
"!data/ilsvrc12/get_ilsvrc_aux.sh\n",
|
||||
"!scripts/download_model_binary.py models/bvlc_reference_caffenet\n",
|
||||
"!python examples/finetune_flickr_style/assemble_data.py \\\n",
|
||||
" --workers=-1 --images=2000 --seed=1701 --label=5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's show what is the difference between the finetune network and the original caffe model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"1c1\r\n",
|
||||
"< name: \"CaffeNet\"\r\n",
|
||||
"---\r\n",
|
||||
"> name: \"FlickrStyleCaffeNet\"\r\n",
|
||||
"4c4\r\n",
|
||||
"< type: \"Data\"\r\n",
|
||||
"---\r\n",
|
||||
"> type: \"ImageData\"\r\n",
|
||||
"15,26c15,19\r\n",
|
||||
"< # mean pixel / channel-wise mean instead of mean image\r\n",
|
||||
"< # transform_param {\r\n",
|
||||
"< # crop_size: 227\r\n",
|
||||
"< # mean_value: 104\r\n",
|
||||
"< # mean_value: 117\r\n",
|
||||
"< # mean_value: 123\r\n",
|
||||
"< # mirror: true\r\n",
|
||||
"< # }\r\n",
|
||||
"< data_param {\r\n",
|
||||
"< source: \"examples/imagenet/ilsvrc12_train_lmdb\"\r\n",
|
||||
"< batch_size: 256\r\n",
|
||||
"< backend: LMDB\r\n",
|
||||
"---\r\n",
|
||||
"> image_data_param {\r\n",
|
||||
"> source: \"data/flickr_style/train.txt\"\r\n",
|
||||
"> batch_size: 50\r\n",
|
||||
"> new_height: 256\r\n",
|
||||
"> new_width: 256\r\n",
|
||||
"31c24\r\n",
|
||||
"< type: \"Data\"\r\n",
|
||||
"---\r\n",
|
||||
"> type: \"ImageData\"\r\n",
|
||||
"42,51c35,36\r\n",
|
||||
"< # mean pixel / channel-wise mean instead of mean image\r\n",
|
||||
"< # transform_param {\r\n",
|
||||
"< # crop_size: 227\r\n",
|
||||
"< # mean_value: 104\r\n",
|
||||
"< # mean_value: 117\r\n",
|
||||
"< # mean_value: 123\r\n",
|
||||
"< # mirror: true\r\n",
|
||||
"< # }\r\n",
|
||||
"< data_param {\r\n",
|
||||
"< source: \"examples/imagenet/ilsvrc12_val_lmdb\"\r\n",
|
||||
"---\r\n",
|
||||
"> image_data_param {\r\n",
|
||||
"> source: \"data/flickr_style/test.txt\"\r\n",
|
||||
"53c38,39\r\n",
|
||||
"< backend: LMDB\r\n",
|
||||
"---\r\n",
|
||||
"> new_height: 256\r\n",
|
||||
"> new_width: 256\r\n",
|
||||
"323a310\r\n",
|
||||
"> # Note that lr_mult can be set to 0 to disable any fine-tuning of this, and any other, layer\r\n",
|
||||
"360c347\r\n",
|
||||
"< name: \"fc8\"\r\n",
|
||||
"---\r\n",
|
||||
"> name: \"fc8_flickr\"\r\n",
|
||||
"363c350,351\r\n",
|
||||
"< top: \"fc8\"\r\n",
|
||||
"---\r\n",
|
||||
"> top: \"fc8_flickr\"\r\n",
|
||||
"> # lr_mult is set to higher than for other layers, because this layer is starting from random while the others are already trained\r\n",
|
||||
"365c353\r\n",
|
||||
"< lr_mult: 1\r\n",
|
||||
"---\r\n",
|
||||
"> lr_mult: 10\r\n",
|
||||
"369c357\r\n",
|
||||
"< lr_mult: 2\r\n",
|
||||
"---\r\n",
|
||||
"> lr_mult: 20\r\n",
|
||||
"373c361\r\n",
|
||||
"< num_output: 1000\r\n",
|
||||
"---\r\n",
|
||||
"> num_output: 20\r\n",
|
||||
"384a373,379\r\n",
|
||||
"> name: \"loss\"\r\n",
|
||||
"> type: \"SoftmaxWithLoss\"\r\n",
|
||||
"> bottom: \"fc8_flickr\"\r\n",
|
||||
"> bottom: \"label\"\r\n",
|
||||
"> top: \"loss\"\r\n",
|
||||
"> }\r\n",
|
||||
"> layer {\r\n",
|
||||
"387c382\r\n",
|
||||
"< bottom: \"fc8\"\r\n",
|
||||
"---\r\n",
|
||||
"> bottom: \"fc8_flickr\"\r\n",
|
||||
"393,399d387\r\n",
|
||||
"< }\r\n",
|
||||
"< layer {\r\n",
|
||||
"< name: \"loss\"\r\n",
|
||||
"< type: \"SoftmaxWithLoss\"\r\n",
|
||||
"< bottom: \"fc8\"\r\n",
|
||||
"< bottom: \"label\"\r\n",
|
||||
"< top: \"loss\"\r\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!diff models/bvlc_reference_caffenet/train_val.prototxt models/finetune_flickr_style/train_val.prototxt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For your record, if you want to train the network in pure C++ tools, here is the command:\n",
|
||||
"\n",
|
||||
"<code>\n",
|
||||
"build/tools/caffe train \\\n",
|
||||
" -solver models/finetune_flickr_style/solver.prototxt \\\n",
|
||||
" -weights models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel \\\n",
|
||||
" -gpu 0\n",
|
||||
"</code>\n",
|
||||
"\n",
|
||||
"However, we will train using Python in this example."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"iter 0, loss=3.786610, scratch_loss=3.163587\n",
|
||||
"iter 10, loss=2.556661, scratch_loss=8.774073\n",
|
||||
"iter 20, loss=2.035326, scratch_loss=2.266603\n",
|
||||
"iter 30, loss=1.943101, scratch_loss=1.703273\n",
|
||||
"iter 40, loss=1.982698, scratch_loss=1.831079\n",
|
||||
"iter 50, loss=1.559268, scratch_loss=2.041238\n",
|
||||
"iter 60, loss=1.464433, scratch_loss=1.836157\n",
|
||||
"iter 70, loss=1.481868, scratch_loss=1.705826\n",
|
||||
"iter 80, loss=1.394870, scratch_loss=1.695532\n",
|
||||
"iter 90, loss=1.055422, scratch_loss=1.867379\n",
|
||||
"iter 100, loss=1.407976, scratch_loss=1.881758\n",
|
||||
"iter 110, loss=1.569579, scratch_loss=1.701803\n",
|
||||
"iter 120, loss=0.951682, scratch_loss=1.764299\n",
|
||||
"iter 130, loss=0.905122, scratch_loss=1.879305\n",
|
||||
"iter 140, loss=1.020678, scratch_loss=1.746009\n",
|
||||
"iter 150, loss=0.784985, scratch_loss=1.739624\n",
|
||||
"iter 160, loss=0.911735, scratch_loss=1.673230\n",
|
||||
"iter 170, loss=0.965255, scratch_loss=1.725484\n",
|
||||
"iter 180, loss=1.028102, scratch_loss=1.676103\n",
|
||||
"iter 190, loss=0.905020, scratch_loss=1.885763\n",
|
||||
"done\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"niter = 200\n",
|
||||
"# losses will also be stored in the log\n",
|
||||
"train_loss = np.zeros(niter)\n",
|
||||
"scratch_train_loss = np.zeros(niter)\n",
|
||||
"\n",
|
||||
"caffe.set_device(0)\n",
|
||||
"caffe.set_mode_gpu()\n",
|
||||
"# We create a solver that finetunes from a previously trained network.\n",
|
||||
"solver = caffe.SGDSolver('models/finetune_flickr_style/solver.prototxt')\n",
|
||||
"solver.net.copy_from('models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel')\n",
|
||||
"# For reference, we also create a solver that does no finetuning.\n",
|
||||
"scratch_solver = caffe.SGDSolver('models/finetune_flickr_style/solver.prototxt')\n",
|
||||
"\n",
|
||||
"# We run the solver for niter times, and record the training loss.\n",
|
||||
"for it in range(niter):\n",
|
||||
" solver.step(1) # SGD by Caffe\n",
|
||||
" scratch_solver.step(1)\n",
|
||||
" # store the train loss\n",
|
||||
" train_loss[it] = solver.net.blobs['loss'].data\n",
|
||||
" scratch_train_loss[it] = scratch_solver.net.blobs['loss'].data\n",
|
||||
" if it % 10 == 0:\n",
|
||||
" print 'iter %d, loss=%f, scratch_loss=%f' % (it, train_loss[it], scratch_train_loss[it])\n",
|
||||
"print 'done'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's look at the training loss produced by the two training procedures respectively."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"scrolled": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[<matplotlib.lines.Line2D at 0x7f39dad72390>,\n",
|
||||
" <matplotlib.lines.Line2D at 0x7f39dad72610>]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": [
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
|
||||
"AAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNXVx/FvD8O+DYsCIgpiRAxE1LiiEY0LGvdEo3Eh\n",
|
||||
"aoxZXkSTGJdouHE3CaAxica4IVETlATEJCohjkrcArIpohFBRFllUUAWmfv+ce4wPUP3dPV09XTT\n",
|
||||
"/D7P089MV1dX3aquPnX6VNUtEBERERERERERERERERERERERERHZbjQBpgMTw3MHLArDpgODC9Ms\n",
|
||||
"ERFJVh5xvGHAHKBteO6BkeEhIiJFoizCOLsCJwL3AYkwLJH0v4iIFIkoQX0UcCVQlTTMA0OBmcD9\n",
|
||||
"QEX8TRMRkWxlCuonAcuwunlyZn430AsYACwGRuSldSIikpVMJZRbgPOBz4EWQDtgHHBB0jg9sQOo\n",
|
||||
"/VO8/12gd86tFBHZscwD9sz3TI6k5uyXbknDrwAeTfMen9cW7VhcoRtQYlyhG1BiXKEbUGIaHDuj\n",
|
||||
"nv0CltVXz+iXwL7h+Xzg0oY2QERE4pNNUK8MD7CSjIiIFJkoZ79IcagsdANKTGWhG1BiKgvdAGkc\n",
|
||||
"qqmLiGSvwbFTmbqISAlRUBcRKSEK6iIiJaQwQd3RpiDzFREpcY0f1B2tsHPbRUQkZoXI1FsCnXE0\n",
|
||||
"K8C8RURKWiGCenUwb1eAeYuIlDQFdRGRElLIoN6+APMWESlpytRFREqIgrqISAlR+UVEpIQoUxcR\n",
|
||||
"KSGFCOrNw18FdRGRmEUN6k2wm09X386uIzAJeAd4FqjIYp4qv4iI5EnUoD4MmENNH79XY0F9L2By\n",
|
||||
"eB6Vyi8iInkSJajvCpwI3IfdpxTgFGB0+H80cFoW81RQFxHJkyhBfRRwJVCVNKwLsDT8vzQ8j6oZ\n",
|
||||
"sA6VX0REYpfpxtMnAcuwevqgNON46r/1kkv6vxIL6stRpi4iUm0Q6WNsVjIF9cOwUsuJQAssEI/B\n",
|
||||
"svOuwBKgGxb403F1nu8NrEBBXUSkWiW1b949vKETylR+uRboAfQCzgb+DZwPPAkMCeMMAcZnMc/q\n",
|
||||
"TF3lFxGRmGV7nnp1meU24FjslMajw/OoVH4REcmTTOWXZM+HB8BK4JgGzrM5Kr+IiORFoboJ+BQA\n",
|
||||
"R4sCzF9EpGQVKqhvAj5B2bqISKwKGdTXoKAuIhKrQmfqOgNGRCRGhQ7qytRFRGJUqKC+ESu/KFMX\n",
|
||||
"EYmRMnURkRKioC4iUkIKdeej6rNfVH4REYmRMnURkRKioC4iUkIKGdQ/BdoWYP4iIiWrkEF9A1Zf\n",
|
||||
"FxGRmBTyPPUNoA69RETiVMhMfSMK6iIisSp0+UVBXUQkRlGCegvgVWAGMAe4NQx3wCLsptTTgcER\n",
|
||||
"56mauohInkS589EG4ChgfRh/CnA4dmu7keGRjeqLj5Spi4jELGr5ZX342wxoAqwKzxMNmKdq6iIi\n",
|
||||
"eRI1qJdh5ZelwHPAm2H4UGAmcD9QEXFaqqmLiORJ1BtPVwEDsL5angEGAXcDN4TXbwRGABeneK9L\n",
|
||||
"+r8S1dRFROoaFB45a0j55HrgM+DXScN6AhOB/nXG9bXm4SgDtmCZfztgIU6deomI1FE7dmYhSvml\n",
|
||||
"MzWllZbAsdjZLl2TxjkdmB1hWk2BTTg8qqmLiMQuSvmlGzAa2wGUAWOAycDDWEnGA/OBSyNMq7r0\n",
|
||||
"AhbUm+Eow1GVZbtFRCSFKEF9NrB/iuEXNGB+NUHd4XEhsFt9XUREctTYV5QmZ+qgM2BERGLV2EG9\n",
|
||||
"+sKjagrqIiIxKnSmroOlIiIxKnRQ17nqIiIxKoagrkxdRCQmhQjqG5OeK6iLiMSo0Jn6RlR+ERGJ\n",
|
||||
"TaGDujJ1EZEYKaiLiJQQnacuIlJCCp2pq6YuIhKjQgd1ZeoiIjFSUBcRKSE6T11EpIQUOlNXTV1E\n",
|
||||
"JEaFDurK1EVEYpQpqLcAXgVmAHOAW8PwjsAk4B3gWWpud5eJgrqISB5lCuobgKOw29Z9Kfx/OHA1\n",
|
||||
"FtT3wm5td3XE+Smoi4jkUZTyy/rwtxnQBFgFnILdt5Tw97SI86t78ZFq6iIiMYoS1Muw8stS4Dng\n",
|
||||
"TaBLeE742yXi/JSpi4jkUZQbT1dh5Zf2wDNYCSaZD4903Nb/ZtKLfZmf9JqCuogIDAqPnEUJ6tXW\n",
|
||||
"AH8HDsCy867AEqAbsKye97mt/+3LvWx7nrrKLyKyo6sMj2rDGzqhTOWXztSc2dISOBaYDjwJDAnD\n",
|
||||
"hwDjI85P9ygVEcmjTJl6N+xAaFl4jMHOdpkOjAUuBhYAZ0Wcn2rqIiJ5lCmozwb2TzF8JXBMA+an\n",
|
||||
"oC4ikkfFcEWpauoiIjEpRFDfnPRcNXURkRgVQ6auoC4iEpPGDupNqZ2pK6iLiMSoGDJ11dRFRGJS\n",
|
||||
"6ExdNXURkRgVQ1BvhiPRyO0QESlJhS2/OHx4rhKMiEgMCp2pg+rqIiKxKYagrrq6iEhMCn32C+i0\n",
|
||||
"RhGR2BRDpq6gLiISk2IJ6qqpi4jEoBjKL6qpi4jEpFgydQV1EZEYRAnqPai54fQbwGVhuAMWYTfM\n",
|
||||
"mA4Mrncqjibh75Y6r6j8IiISkyj3KN0MXAHMANoA04BJ2M2mR4ZHFKlKL2Dll5YRpyEiIvWIEtSX\n",
|
||||
"hAfAWuAtoHt4ns3l/alKL2CBvmkW0xERkTSyran3BPYDXgnPhwIzgfupuUF1OgrqIiJ5lk1QbwM8\n",
|
||||
"AQzDMva7gV7AAGAxMCLD+9OVXzaH10REJEdRyi9gmfQ44E/A+DBsWdLr9wET07zXATCC9pyWciey\n",
|
||||
"CQV1EdmxDQqPnEUJ6gmsvDIHuCNpeDcsQwc4HZid5v0OgB/TGzg1xeubUflFRHZsleFRbXhDJxQl\n",
|
||||
"qA8EzgNmYacuAlwLnIOVXjwwH7g0w3TSlV+UqYuIxCRKUJ9C6tr7P7OcV7oDpcrURURi0phXlNZ3\n",
|
||||
"9osydRGRGDRmUK/v7Bdl6iIiMVCmLiJSQoohqCtTFxGJSTGUX5Spi4jERJm6iEgJKYagrkxdRCQm\n",
|
||||
"xVJ+UaYuIhKDYsjU1aGXiEhMiiGoK1MXEYlJMZRflKmLiMREmbqISAkphqCuTF1EJCbFUH5Rpi4i\n",
|
||||
"EhNl6iIiJaQYgroydRGRmEQJ6j2A54A3gTeAy8LwjsAk4B3gWaAiw3R09ouISJ5FCeqbgSuALwKH\n",
|
||||
"AD8E+gJXY0F9L2ByeF4fdRMgIpJnUYL6EmBG+H8t8BbQHTgFGB2GjwZOyzAddeglIpJn2dbUewL7\n",
|
||||
"Aa8CXYClYfjS8Lw+6npXRCTPsgnqbYBxwDDg0zqv+fCojzJ1EZE8K484XlMsoI8BxodhS4GuWHmm\n",
|
||||
"G7AszXsdABM4gI4sSfG6MnUR2dENCo+cRQnqCeB+YA5wR9LwJ4EhwO3h7/ht3wpUB/VT6QvMYvI2\n",
|
||||
"rytTF5EdXWV4VBve0AlFCeoDgfOAWcD0MOwa4DZgLHAxsAA4K8N0dPaLiEieRQnqU0hfez8mi3nV\n",
|
||||
"X1N3JHAZ6/IiIlKPwvf94qgCqoAmjdgWEZGSVAzdBIDq6iIisSiWoK66uohIDApffjHK1EVEYqBM\n",
|
||||
"XUSkhBRLUFemLiISg2IpvyhTFxGJgTJ1EZESUixBXZm6iEgMiqX8okxdRCQGytRFREpIsQR1Zeoi\n",
|
||||
"IjEolvKLMnURkRgUU6auoC4ikqPGCeou9MDo2JJmjE2o/CIikrPGytTrK72AMnURkVhECeoPYPcj\n",
|
||||
"nZ00zAGLsDshTQcGZ5hGfaUXUKYuIhKLKEH9QbYN2h4YCewXHk9nmEamoK5MXUQkBlGC+ovAqhTD\n",
|
||||
"E1nMJ1P5RZm6iEgMcqmpDwVmAvcDFRnGjVJ+UaYuIpKjKDeeTuVu4Ibw/43ACODiNOM6RtGBfrQB\n",
|
||||
"BgGVKcbRxUcisiMbFB45a2hQX5b0/33AxHrGdVxBX+B4/pMyoIMydRHZsVVSO+Ed3tAJNbT80i3p\n",
|
||||
"/9OpfWZMKlEOlCpTFxHJUZRM/THgSKAz8AG2BxkEDMDOgpkPXJphGqqpi4g0gihB/ZwUwx7Icj5R\n",
|
||||
"Lj5qneU0RUSkjsa6olSZuohIIyiWoK6auohIDIql7xdl6iIiMVCmLiJSQoolqCtTFxGJQbGUX5Sp\n",
|
||||
"i4jEQJm6iEgJKZagrq53RURi0FhBvQ2wtp7X1fWuiEgMGiuotwfW1PO6MnURkRgUS1BXpi4iEoNi\n",
|
||||
"CerK1EVEYlAsQV2ZuohIDIolqCtTFxGJQbEEdWXqIiIxKJagbpm6ozmOro3UJhGRkhMlqD8ALKX2\n",
|
||||
"Les6ApOAd4BngYoM04iaqV8N/CFCm0REJIUoQf1BYHCdYVdjQX0vYHJ4Xp8omXor4HtA3whtEhGR\n",
|
||||
"FKIE9ReBVXWGnQKMDv+PBk5L+25HUywLX1/PPDZhgX8esBtOB01FRBqioTX1LlhJhvC3Sz3jtgc+\n",
|
||||
"weHrGae6X5iRwEKgdwPbJSKyQ4ty4+lMfHikNorr6UcZ4IDK8KhrAzAKeBK4EOgDvBVD20REtgeD\n",
|
||||
"wqPR9KT2gdK5sPUslW7heSoex/44pkeek+NXuIw1ehGRUlZfZaNeDS2/PAkMCf8PAcbXM26mg6R1\n",
|
||||
"vY1l6iIikqUoQf0x4CUs0H6AlUduA47FTmk8OjxPJ9ugPhfYO4vxRUQkiFJTPyfN8GMizqNhmboj\n",
|
||||
"keHgqoiI1NEYV5RmG9RXYPWknXD0yE+TRERKU/EFdcvO3wZeBubrnHURkeiKL6ibEcA1wMdAp9hb\n",
|
||||
"JCJSooozqDvG4RgLLAc656NRIiKlqDiDeg0FdRGRLBR7UF+BgrqISGQK6iIiJWR7COo7xdgWEZGS\n",
|
||||
"tj0EdWXqIiIRKaiLiJSQxgjqzYF1DXyvzn4REclCYwT1NTn04aJMXUQkC40R1Jfk8F4FdRGRLGwP\n",
|
||||
"QX0nHIm4GiMiUsqKO6g71mM9NraKrTUiIiWsMYL64hzfrxKMiEhEud54egHwCbAF2AwclGKcXMov\n",
|
||||
"UHMGzPs5TkdEpOTlGtQ9dgfslfWMk2tQV6YuIhJRHOWXTAcxFdRFRBpJrkHdA/8CpgKXpBlHQV1E\n",
|
||||
"pJHkWn4ZiB0I3QmYBMwFXqw1xu1cAKwPzyrDIxvq1EtESt2g8Cgqw4Ef1xnmcTn+GnB8H8c9OU1D\n",
|
||||
"RGT70tCr8HMKuK2AtuH/1sBxwOxtxnJU5TAPgGVA1xynISKyQ8il/NIF+FvSdB4Bns25Rdt6C9gn\n",
|
||||
"D9MVESk5+b783uc8D0dTrOvenXAN7u1RRGR70uDY2RhXlObGsRl4G2XrIiIZFX9QN7OALxW6ESIi\n",
|
||||
"xW57CeqzUVAXEcmoEYK6j2Mes4D+OU/F0V3d+IpIKWuMTL1fDNOw8ksuAdnRDXgP+HYM7RERKUq5\n",
|
||||
"XlEaxVexoJwFfy/wFiRGhQFLsaPBZ+G4HtiEnRHTArui9TXgLzjm1zPRoViXBr/EMQXH/7Jrk4hI\n",
|
||||
"8WuEUxr9U5A4OYu3nA7cBnwOPA38BBIex2RgP+AiYBHQjnU7baHlyl0o2zKQqrJzWLP7fDrMP2eb\n",
|
||||
"gO1og3UTfDAwGOun5ggcn+a8hCIi8cv9dPA88eBXg28Xnp4Mvks9o1eA/wD8V8B3AD8TvHUU5tgf\n",
|
||||
"R6+kcfuBfx/8TwFovvpOBt62keGJj3HciAt3S3LsjOMuHI+H5wkc9+J4FkezMOxQHONwNIl5+UVE\n",
|
||||
"GqLB3QQ0Rqb+ODAF+AswH7gDEteGl5sBD2BdDszGMug/QeKn4fW9sQ7CrgXOAaYD1wNnAr8C7sEy\n",
|
||||
"973DtN9kl9f+w3cP7o11jrMBn+jA/KOWMeXqh3jvuJsBcJTzSfcXSFStpO3iM4DXsS4P7sQxMuel\n",
|
||||
"dlQAXwMexTX8w4kwn5Y4Psvb9PPF1s/ZwB9qrR9H67xeYOboCkwAvo3jrTzO5zDgNRyf520eUuoa\n",
|
||||
"nKk3RlA/HHgIeBI7g2UfYHegCngYaA+MA74MPACJ6XUmMQT4P+BO4FTgBOANG5aYCn4ydveldsCV\n",
|
||||
"2Jd2Dy4+rB+Tbj+XDw8cwpYWfwrv/SMwEzifpmsPZsjRPahYsJjPW67k778fy7dO+hEJbgL2BF4A\n",
|
||||
"JuD8TpSvr+K61vsBc7bW7e2g7ReBj3CsxLEz1gfOFmBiWMaf4rgX/DeAFyGxFEc7oAzHahxtsR3S\n",
|
||||
"tLR95Nh8zgAuBm7C8VIY/mU8U0hwAY6xOPoD63C8l+VnVD9HR+x00iocL8Q0zYexZboDx3Vh2FBg\n",
|
||||
"BJ91eICp3xvDEbe+gmNLmvcnsM97U9qdmqMlcC7QHJiK41UcfwjL0hU4DJfjrRYdzYEfAAuBf+NY\n",
|
||||
"hePrwFjgUeygfFdgM45lOc7rCOB0bBtYmTQ8kTFxsPXVDcdHGcarwBKrx3AswrEX8Aku5+6zk+fR\n",
|
||||
"EfgJcBeOxTiOB1bgmBbx/e1DmzItcwtgY52koRXwWayJlq3b3YH3cfjwa78qhnkUdVAvA/6LBcAv\n",
|
||||
"YP3FXEdNV5NfhcT6dBOoM7kEFgTfhkQIgv5kbIfxLUg8Br4SW8mdsODqIPE/8LthO5GNwEvAr+g7\n",
|
||||
"bgAH/fYpnhk1lSUDYP8/7sNXbppBxcLngcF49uGF6zbTZVZn9nx6MWVbOvJptxW0Xl5J+cb+QHds\n",
|
||||
"p7QOaIntXLoCvwCeAF5kza4T+LjPxbRbtI5Oby/H+sypIsGHwC7UHASegHVR3B3ogd2+bzNwILaz\n",
|
||||
"uA+4Kvy9hc0tpvHa0B4cdNcmmm6YAfTBDnyvDtP8GOsKeTnQBNtpNgdGUVXWitU9L6LD/MdJ+Bd4\n",
|
||||
"7G+30Ou5jznkNwuB3YCmYR19BTgP24l2x7pXvgnrZK36s7gE6xp5QWhDJ+D3wEfY8YtdsAPbj2IH\n",
|
||||
"tE8K6+dY7MD1S2F632T53hfw0QHj6P2vClov20zC/xm4N6yPHkAb4DDgcNiaBVcC44GncCwLX7ID\n",
|
||||
"gAeBD0K7vh7a9MPQxkuBy7Bfe+8C3cIyl4Xleg+oXqd7YzuQRPh83gbmhXV5D/AplqAcDIwO6+sM\n",
|
||||
"4MawzjqF9T8deDl8HvuGec3DfiHulDTvKizxeCvMu3tYh0eG9XV4WJYewBGhjU+F7WV/bFtaCryD\n",
|
||||
"HXtqAnwXS6geBm4Ny/F5WOZ9QhuWYtvWh9i28gG2rbYIr60I71uHHet6BNtpHYJ9vyuwbbVjWCcT\n",
|
||||
"se/GIGBOWJ/lwM9C23pjv96HAM2w7WgdsFdYxx8An2FxYxfgP2G9Xg/8L6yDZ4EBwI/CPP4ZluV4\n",
|
||||
"4DTs+zMN+27tin3+C4G/Yzv39eEz3DUs89/CejwmLHszYC12UsZq7ISPWTiqQul2X+Dm8Jm8ht1X\n",
|
||||
"4kIsDjwd1sOq0PbWWPybgOP9sHNri303mofXWwELcVtjQrEGdRLgTwAGQOJW8D/AgtMGYCAkVuQ4\n",
|
||||
"izLg58BtkNgAvj32IX0EiVVZTCcBfAP4LTASGMGXHr6Gfn8eRvO1MxnzzABgC3tPWEC7RXuxucXr\n",
|
||||
"TLv0Q9ovvIFhe36O7amrcJRtzbodRzPzvN+xrP9/+fDLPdnQoRnL+vUGv5ozz/oBMy5cztsnf4d9\n",
|
||||
"ntiDAaM/Za9/zLV2s4g5Z5zG8n2O4LOOzzH9wtfZWLGRYXtMpsP8e/F8mUUHN+X+l/7Czm98g/0e\n",
|
||||
"fInXLxnOZxUfc/R1/Rnw8FrKtnTBgkKFfQbMwCdaUFV2Jat6d2LmBR04dNR/aflxb5b1q+CzTmvo\n",
|
||||
"+fxjwEKqyhIkqgaS4B3gNhzLcbTDJ0aBP54EO2PB5yMseL7L5817U77xbWyneQXQBs/feeXyvqzv\n",
|
||||
"fDRH3rCc8k27Ayt4zt3J88P7cFWHG2i5+gxW9j6SyTe9yJtnfxULIDNpueJyhvZ5nFYrT8K+xPOx\n",
|
||||
"L+E04Dkca3B0AE7EfoUdj+3IEmwpb4pv4ijfeH/InvbBvmS/wnFX+Gz2o6rsKsqq2oXl2EhNHbMP\n",
|
||||
"PrEfCf8WtkNbE5a3DRYEe7K5ZQvWdvkrHRZcHj733bBkpRLHo+Hg/MFUXv8a3aZ7+jw1CAu6O2PB\n",
|
||||
"YTMW2D7BAv2WMP9yLDj2Ad7EgtA6YCyOj8MvgSPD8JexndJp2I5hWmhnt/D+blhQnwD8C8+twMkk\n",
|
||||
"KMcCehW281iL/Tq9BxiF7TB6Y79WE2GZ22FBpwL71XgUMBkLhPtjQe+VsCxdwmeyJoyzd5hGAhiH\n",
|
||||
"YwKOS7BfUudh39fHsUD+Jhb8dsEC3VzsRjtHhLZeEpbtQmyHsQS4JbT/K9itNadiO7AyLAk4Iyzj\n",
|
||||
"rUAv7Iy818P6uQTbtl7Hvv87Y8lLdVLVJix7h/C5dA/bSqswzr3YDuZcbIf0RyzBOxoL1t2w+058\n",
|
||||
"Gj6zU8K62in8bYbFwvXh8QscEynuA6XbDOpo2bTvte1rxcDvBv5f4BeAXwb+wDB8v6QDvgeAvwj8\n",
|
||||
"7eBng+8Ofjz4VeCngj80jHcg+MXgW4cDv78Avyf4n4BfCH5FGPadML9Lw/s6g18C/ofgfwf+AfBT\n",
|
||||
"wD9D2cZ+HHPVY3SYNwt8kzC9e8DPD/NaDH5amGYH21n5L4B/DPwm8J+Cvwv8teAfttKQfwH8UvC9\n",
|
||||
"wZ8F/nPwPhyovgL8hWFZF4eD0yPB3wr+1+C/B/6hMO1nwnq5D/zE8DlPBX8T+DfAnwr+kbCsb4I/\n",
|
||||
"F3zfsB4eB38v+OZhHYwE/5TttH05+KZ1Pqdy+5Xm/wz+bso2fpv+f7qcnv/+K2zZDP6WWqM7mrL1\n",
|
||||
"Ogd/HPhXwG+m+kB77WkPDOtgXPiFV/f13UOb37V1XOu1tvY5+h+BvzNsE3/IvN1tff+5WGKSBd8k\n",
|
||||
"xfrpA/7gOuM8AH4u+K+FbaNNdvPZOt02uHzeuMYn7LHN8DLwDnzuFyI2lKM9js6hvNeQ97fF0Y/M\n",
|
||||
"J2Xk71hcjoq2YZn5g8Cfn2GcBPg/gt8Ygm8X8N8MAfKH4e9Zad57pgXkrc97Y4H+IfB/BX9HnfHL\n",
|
||||
"wY8Iwftp8Gm6TfBl4E8MAekT8OtCALqu9pfYd8LOTKoMgfg3Nk+/2IKBLwN/TAjQD1mQ9H1tvv56\n",
|
||||
"8D8Df1V4/Wdh2S8B/wT4y0IAPxt8q7CebsR2HleCbwf+cGwH8TL2663ucjQNr40B/15Y7uqd5V7g\n",
|
||||
"XwuP6gD6KPixYT59wS8Pf4eF8aaA/2XS53IqNTuU3knz7YDtdM4EPxz8Smzndz62M9gX/PNh2e8I\n",
|
||||
"n0V/8L3COl4S1sFI8D8P6+tD8APr35YA/Dngt7B1J+B/ZI+tr+8J/vvYTrZ1GNYE/ARsJ7l7GNYT\n",
|
||||
"/KKwnD3ANwvLMBn86dgOdlXYPpJuQ+mPDuupS9K21NzmAeCPB78e/G8zL0vGZT0e28EcXGd4P/DT\n",
|
||||
"sR16nZ2Ovzy8Zzn4i9km8Pvm4B8M679ThvmfBv6M3JcjCt8afHmK4W3DNvvFVG/Ke7MaqGgbFh/f\n",
|
||||
"DPyRdYZ9PQSsI1O/J+20OmIZ9GT7wHNuW+v6p+MfBh9KVv5ALDuP4QsbuX1PgH+RtF1J+N2wTPyI\n",
|
||||
"EISXhiD8Cfj/2/ZLXeu9l4XANc0+B38k+NtC0EraIfqfYkF/cpj2SvC/SXq9A7YjegT7BTcrfBFD\n",
|
||||
"duzvAv9OeO/vSZlF+rNC0B0SHg779TIUSx5agj80BKujQkD+Hfh52Cm+J4O/Gfvl+FAI4ivD/B4E\n",
|
||||
"PwnbAXwU2vlemPbV2C+SadgvyVZ12tUH/Bxsx/4gtkMai/3K2j28tgn7RTMvrP9TwrLuYevRf4et\n",
|
||||
"v65qTfuLod3fwHa8Z1Ozc0iAfym0fxmWGJ0N/v6wDr6DJQsz7bPwt2IBfTmW/PTFdkxjwnZ7MvaL\n",
|
||||
"cUrYpkZhCdLhYV5XYInSCGzH++fwmS3Ggvve2C/mieB/y9YdJmCB93Lw/wX//TDsEFLuEHxZaN++\n",
|
||||
"4XEUts2tCevyVGpt6/6esB6WYknDCeB3rX4x9XadWa41m8HAHVjd7j7g9jqvF21dSAB8P+DkcKwj\n",
|
||||
"gR2buAMSaxpp/q2Ackh8ksX4rYF1mQ+u+3LsQN4jkKjntE/fFHBYHfZlrNa8CBIxJiQ+gR1H6ovV\n",
|
||||
"ed/H6rK7UHOwcx5wOyQetCDFvVjtvDPwb+xg5BmQWB6muSt2Om8/4BL7zPyBWF13FST+EQLIfcCr\n",
|
||||
"Nr1Uy+RbY/Xf3tgJB4vCetjT1ktiVFhHewBrIfEh+J9jp+zugZ2KvBt20PlZLCZchJ0wMAOrS6/A\n",
|
||||
"Dni2xq7sToR29Q3jnRPaUAmMhsRSao5x7YIdUNwTGAeJiaHdrbBjX4dgB3c/DPO7x06i8CeFeSzE\n",
|
||||
"jh3cidXuK2w5+A1W65+EHawcE5b7VOAg7GD+IuyY0UzsQP8I7IDooLBcN2PHO4aF6VbXyVeHZVwZ\n",
|
||||
"3vvr8DndjNXmx4XXvosdsD0QOw7VBPv+PU2BYmcT7CBNT+yLMAP7kJLtAJl6oxlU6AaUmEGFbkCN\n",
|
||||
"VL9UfFLN1fdPnQ3nrT29wH+tntfbgn8S/AHh+THwl+ewX0ZjwQ+u3X4IGfOZ2K+QxZaN55vvDv67\n",
|
||||
"2PUw6cbptO269WeA/yf2y+WbScN3C78meoaMfAFWIjwGK411zNCeBHZsrvp4Vn0luYLEzkOxMwqq\n",
|
||||
"XR0eyRTU4+MK3YAS4wrdgBLjoo3mK0I5pBF3UvmS7oBuPBNv6Btz6aWxO3YKUrVFYZiISBqJ1VbS\n",
|
||||
"SWwsdEtyl/DxlunikUtQL7qFERHZ0aU4zSayD7GLFKr1wLL1ZPNQ8I/T8EI3oMRofcZL6zM+8wox\n",
|
||||
"0/Iw457YVVGpDpSKiMh25ASsn4Z3gWsK3BYREREREclkMNYRz/+wiy4kewuwjp+mYxc8gPWANwnr\n",
|
||||
"5e5Z7IIHSe0BrHOw2UnD6lt/12Db61zguEZq4/Yi1bp02DG06eFxQtJrWpf16wE8h3Ve9gbWYygU\n",
|
||||
"8fYZ5aIkyWw+9iEn+yVQ3QHVVdht/yS1I7DbHyYHonTrbx9sO22Kbbfv0jg3Zd9epFqXw7Eub+vS\n",
|
||||
"usysK9ZlMFgvkG9jMbJot88oFyVJZvOxvriTzcW6NQXbMOY2aou2Pz2pHYjSrb9rqP2L8mns8nOp\n",
|
||||
"0ZNtg/qPU4yndZm98Vgf7rFsn/mI9rooKR4euynCVKzPZ7APfGn4fyk1G4BEk2797ULt03G1zUYz\n",
|
||||
"FOvb5H5qSgVal9npif0KepWYts98BHWdlx6PgdiHfQJ2x5Yj6rzu0brORab1p3Vbv7uxG04MABZj\n",
|
||||
"nV2lo3WZWhusc69h2E00kjV4+8xHUI9yUZJkVn3/zOXY3WUOwvbeXcPwbpDjfS93POnWX91tdtcw\n",
|
||||
"TNJbRk24i3YpAAAA0klEQVTguQ/bPkHrMqqmWEAfg5VfIKbtMx9BfSp2O6qe2EVJ38S69JToWmHd\n",
|
||||
"jYJ1V3ocVs98ErunI+Hv+G3fKvVIt/6eBM7Gttde2Pb72jbvlmTdkv4/nZp6u9ZlZgmsZDUH67q8\n",
|
||||
"WlFvn7ooKTe9sKPdM7BTnqrXYUeszq5TGjN7DLv36CbsGM+F1L/+rsW217nY/U6lRt11eRF2D9BZ\n",
|
||||
"WE19PLWP72hd1u9wrI/3GdScEjoYbZ8iIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIikk//DzX8Jz0M\n",
|
||||
"ra0pAAAAAElFTkSuQmCC\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"<matplotlib.figure.Figure at 0x7f3a2803d590>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(np.vstack([train_loss, scratch_train_loss]).T)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Notice how the fine-tuning procedure produces a more smooth loss function change, and ends up at a better loss. A closer look at small values, clipping to avoid showing too large loss during training:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[<matplotlib.lines.Line2D at 0x7f39d50acc90>,\n",
|
||||
" <matplotlib.lines.Line2D at 0x7f39d50acf10>]"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": [
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAXgAAAEACAYAAAC57G0KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
|
||||
"AAALEgAACxIB0t1+/AAAIABJREFUeJztnXe4JGWV/z81Odw7OScmz5DzEEQYdJUgAioqqGsWTAs/\n",
|
||||
"d1dXV11rjWvOAQMKiJhFRERQHBRRchgYZpicmByZfGemfn+c9731VnWlvre6+/blfJ7nPrdDdVV1\n",
|
||||
"dfe3Tn3Pec8LiqIoiqIoiqIoiqIoiqIoiqIoiqIoiqIoiqIoiqIoz2t6Ao8Cv0t5/mvAYuBx4MR6\n",
|
||||
"7ZSiKIqSTo+Cy10DLACChOcuBKYDM4ArgW+Xs2uKoihKZygi8BMQEf8+4CU8fzFwvbl9PzAEGF3K\n",
|
||||
"3imKoigdpojAfxl4P3A45fnxwGrn/hrkpKAoiqI0kDyBvwjYiPjvSdG7Jf5ckpWjKIqi1JFeOc+f\n",
|
||||
"iVgwFwL9gEHADcAbnWXWAhOd+xPMY3GWANM6vKeKoijPT5Yiec6acg7JVTQXAreb26cD/0x5fYDP\n",
|
||||
"Snwmd3gPfAbgs6fDr+9e+I3egW6E3+gd6Gb4jd6BbkaHHZG8CD5tQ1eZ/9ci4n4hEqHvBt6S8fp+\n",
|
||||
"wL4qt+nSBvTuxOsVRVGeN1Qj8PeYPxBhd3lvwXV0VuAPAr3w8fDV51cURcmiaB18WXRO4EXUDyED\n",
|
||||
"r57vzGv0DnQj5jV6B7oZ8xq9A4pQb4HvA+zv5DrUphHmNXoHuhHzGr0D3Yx5jd4BRai3wO8vwVpR\n",
|
||||
"gVcURSlAvQW+M/675SAq8IqiKLk0o8C3UX31j6IoyvOOZhV4jeAVRVFyUIFXFEXppjSjwKsHryiK\n",
|
||||
"UoBmFHj14BVFUQrQrAKvEbyiKEoOKvCKoijdFBV4RVGUbkozCrwmWRVFUQrQjAKvSVZFUZQCNKvA\n",
|
||||
"awSvKIqSgwq8oihKN6XeAr+3hHWoB68oilKAZo3g1YNXFEXJoVkFXiN4RVGUHFTgFUVRuikq8Iqi\n",
|
||||
"KN2UZhT4g6gHryiKkkszCrxG8IqiKAUoIvD9gPuBx4AFwGcSlpkL7AAeNX8fSVmXCryiKEqdKGJ1\n",
|
||||
"7APOBfaY5e8FzjL/Xe4BLi6wrs6iAq8oilKAohbNHvO/D9AT2JqwjFdgPdpsTFEUpU4UFfgeiEWz\n",
|
||||
"AfgLYtW4BMCZwOPA7cBRKevRgU6Koih1oqhQHgZOAAYDf0Q893nO848AE5FI/wLgFmBmxVq+zpvN\n",
|
||||
"85jXz6tYJp82YEAHXqcoitIMzDV/DeGjwH/mLLMcGBZ7LMBnVqe37vMBfD7X6fUoiqI0B0FHX1jE\n",
|
||||
"ohkBDDG3+wMvQSplXEYTevBzzO0kn149eEVRlDpRxKIZC1yPnAx6ADcCfwauMs9fC1wGvAsR3z3A\n",
|
||||
"5SnrUg9eURSlThQRyvnASQmPX+vc/qb5y0PLJBVFUeqEjmRVFEXpptRX4Od97EAJa1GBVxRFKUCd\n",
|
||||
"Bd4vQ5i12ZiiKEoB6m3RlCHwGsEriqIUoN4C36eEdajAK4qiFEAjeEVRlG5KMwq8DnRSFEUpQDMK\n",
|
||||
"vA50UhRFKUCzCrxG8IqiKDmowCuKonRTmrGKRj14RVGUAjRrBK8evKIoSg7NKvAawSuKouSgAq8o\n",
|
||||
"itJNUYFXFEXppjSjwGuzMUVRlAI0YxWNRvCKoigFaMYIXgVeURSlAM9fgfd5HT6zO787iqIoXZN6\n",
|
||||
"e9ldqdnYa4C+wMIS1qUoitLlaNYIvowTU39gUAnrURRF6ZI0o8AfAnrg43VyPSrwiqJ0a5qvisYn\n",
|
||||
"oBybRgVeUZRuTZ7A9wPuBx4DFgCfSVnua8Bi4HHgxIz1lVX9UpbADy5hXxRFUbokeV72PuBcYI9Z\n",
|
||||
"9l7gLPPfciEwHZgBnAZ8Gzg9ZX1lCXwZPrxG8IqidGuKWDR7zP8+QE9ga+z5i4Hrze37gSHA6JR1\n",
|
||||
"lSnwatEoiqJkUETgeyAWzQbgL4hV4zIeWO3cXwNMSFmXCryiKEqdKGJzHAZOQPzqPwJzgXmxZeIV\n",
|
||||
"LUHyqq44F/DNnXkJ6ylKWQKvHryiKF2Nueav01TjY+8Afg+cQlSY1wITnfsTzGMJ3PwA/NSvZgdT\n",
|
||||
"6FzDMZ8eyCAnjeAVRelqzCOqsR/r6IryLJoRiKcOEvG+BHg0tsytwBvN7dOB7Yidk0RXsWj6mf8q\n",
|
||||
"8IqidFvyBH4scDfiwd8P/A74M3CV+QO4HVgGLAGuBd6dsb6uIvD9geeA1hIGTCmKonRJ8myO+cBJ\n",
|
||||
"CY9fG7v/3oLb62oC3xMYAOwuY6cURVG6Es3YbAw6P9CpP7AXuYIZjAq8oijdkGbsRQOdH+hkBX4H\n",
|
||||
"6sMritJNqXcEX8aMTlCORbPX/KnAK4rSLWmCCD7wIIjvpwi8z0x8BnZgP6zA70QFXlGUbkoTCDxv\n",
|
||||
"BT4Xe8x68D8AbuxAJYwr8DrYSVGUbkkzJFnHAFNij1kPfhzSKuFb+MwEluC3l29moR68oijdnmaI\n",
|
||||
"4FuAUbHHrAc/FngZMnr2Z8Br8RleYJ1q0SiK0u1phgi+lWSBHw4cxudp4OUA+MwFXo/0p89CBV5R\n",
|
||||
"lG5PM8zolBbBTwLWxR7/HvD2Ap68evCKonR7msGiaQWGQNDXeewg0uAsLvD3ICNTk0bfuqgHryhK\n",
|
||||
"t6cZBL7F/B/pPNZGksD7HAbuAF6Ys061aBRF6fY0g8C3mv+uTZNm0QA8Apycs04VeEVRuj3NIPAt\n",
|
||||
"SPvhuMAnWTQAD1OdwKsHryhKt6QZBL4VWEp0nteDyIQd6xOWXwAckTPCVT14RVG6Pc1SRbOUygge\n",
|
||||
"kiJ4nzbgKWSawTTUolEUpdvTTBF8MYEXHiG7kkYFXlGUbk8XF/igNzIYayXVC3yWD28FXmd1UhSl\n",
|
||||
"29LFBZ4WYBeSZI178G3AlpTXPUyRCN7nILCHsFJHURSl29AsAr+Rygh+PT5ByuueBGbhp7ZisBE8\n",
|
||||
"wFZgWJX7pSiK0uWpt8ADQc8qFm5FbJQkgU+zZ8BnPxL1T0xZwhX4bajAK4rSDam3wB+gukoaG8Fv\n",
|
||||
"AkbK5B9AnsATeGyftJ/KNsOWeAQ/tIp9UhRFaQrqLfDVTrVnInhvHyLIQ8zjS4G/Z7xuICvmTiPw\n",
|
||||
"pqY8rxaNoijdnnq3C65W4G0ED6FNsw2fO5CeM2n0Y9s0j8O9poUFNxFU4BVF6fYUieAnAn9BBg89\n",
|
||||
"CVydsMxcZFToo+bvIynr6mAED1T68Fn0Y9sUCHpMT3leBV5RlG5PkQi+DXgf8BgSUT8M3AU8HVvu\n",
|
||||
"HuDiAuuqNoK3Ar+d0KLJox/bpwBBpQcvNe/xJOvIiuUURVGanCIR/HpE3EHskqeRuVDjFBks1JEI\n",
|
||||
"3lo01fSNkQi+x6EjEp7rjcwEddDc1wheUZRuSbVJ1snAicD9sccD4EzgceB24KiU13ekisZG8Dmd\n",
|
||||
"H4NpELzI3OnLrrHgHW7FZ0BsQTd6B62iURSlm1JNkrUF+CVwDWFUbXkE8er3ABcAtwAzK1fxn0Ph\n",
|
||||
"d1cjfvo885dFK7DW3M7rG3MRcCpwN9CPoAcc7Lue3vsmIx0mLUkCrxG8oihdhbnmr9MUFfjewK+A\n",
|
||||
"HyPiHec55/YfgG8hork1utgX1sAXrgPvoYLbdSP4PItmNCLeAP0AONCynt77pqICryhK8zCPaPD7\n",
|
||||
"sY6uqIhF4wE/QETyKynLjCb04OeY21sTltuLFd9iuB583uQco511y/+9wzdQOdgpLvCVI1l9zq5i\n",
|
||||
"HxVFUbokRQT+BcAbgHMJyyAvAK4yfwCXAfORZOxXgMtT1lWtwFcTwY8hHsHvHL+FfIGPRvBSZXM3\n",
|
||||
"vrYRVhSluSli0dxL/ongm+Yvj32EIlyEaiN4O6pJBH7r9K1MvXtabLm4wO8BeuLTD5995rU9zbZ3\n",
|
||||
"VrGviqIoXYp6tyrYS3UC31EPvi8A60/cQV4ELx0p3UqaFvNfWwgritLU1FvgbYQMBD3kL5OCEXzg\n",
|
||||
"IaNcoxbNinOeA6bEJvSIR/AQtWlaY/8VRVGakkZG8O8lvaWBJV4HnxbBD0Hq66NJ1i0zbb94t859\n",
|
||||
"OHI14OImWjWCVxSlW9BIgR+JiG0W8ZGsaR78GKSVQTSCD3r2A5YBblfJU5B2Cy5uBN8S+68oitKU\n",
|
||||
"NNCioT+ZFTWBBwwkatEMcnrCu4wGVhAV+MDcX07Uhz+dypG46sEritLtaGQEPwCbDE1mALAPvENy\n",
|
||||
"19sPHCb5pJAk8DaiDwVe2hbMRko9XdSDVxSl29HICH4A2TXxg4mOkIV0H340sEZuBr1IE3iZiHsB\n",
|
||||
"fmaSVSN4RVG6BV05gp+KROUuaT78GGQOVrv+vkjitB9RgT8N+GfC6zXJqihKt6PRAp8Vwc8EFsUe\n",
|
||||
"y4rgXYHvh4h2PII/jUr/HSoj+INoklVRlCan0RZNVgQ/C3gm9lhaLbwVeDtS1rVoVgBH4NMTSbAm\n",
|
||||
"RfBukrUV6YGvEbyiKE1NF4vggzEQ2JLGWVRG8GY0a3AmBK93Hh+NiLIbwW8H+uGzB4nmvwhsQibs\n",
|
||||
"jhOP4NehAt/98bkNn5MavRuKUisaIfBZEfxbgM+a2zNJj+AvQhqcWVyLph9RiwbEprkcuBQ/GArB\n",
|
||||
"CbH1xgX+WVTgnw9MACY1eicUpVY0wqKxoptUBz8MmAtBbyTJujj2vO1HcyQywQimLj7LgwdphPZy\n",
|
||||
"fFYDrwI+EVuvm2RtRSP45wsD0dm8lG5MNTM6lUHcookPWhoKjAAuBNaDFy9ntEnWIwl/mMOAPeDt\n",
|
||||
"g8CtoglHtvrc5KxjitmGyw6g1fj0LUjv+9M68P6U5mIgOtmL0o1pdJI1KYLfDLyTSnsGRIhHIHPD\n",
|
||||
"DoWgLzANaUdg1+9G8ElVOlORNgkhPoeQk8cQ1IOvPz5nxBrC1QsVeKVb0+gka9yDHwb8FjiPygQr\n",
|
||||
"iAifBKxGfPLxiMDbxKnrwbu9aVwSIvhgCvtbDyBXBeV78D7vwa9qopPnG7dS2da5tsgJRS0apVvT\n",
|
||||
"IIEPehHt/mgZBvwasW6SIvidwMnA04jIT6RS4JM8eJepwGDj81tezI6JA8z2a+HBf5JowzMlyiDq\n",
|
||||
"H0n3QSZ20Qhe6bY0yqLpj8y+FI/ghwJPIuL+dMLrdyCR/0KkNcEERDitRRMvk4wJfNBqXr+JaCfL\n",
|
||||
"I9gzogfyY28BNiIllj2rf4sxfPog1k/WbFTPX3z6ImJb70h6oPlfW4H3+R+d41dpFI2yaAYgAtw7\n",
|
||||
"NunHMKRk8cXA3Qmvt1PopUXweR78FGTg0yaiPvwkdo/qQyjwO5Gp/AbSeex2hpSwru6IvVKqdyRd\n",
|
||||
"H4GXwXXH1ngbipJIoyL4AcBuYD/tUXzQD+gtj3trwDuc8Ho7UYcV+Akke/B9kUZlPSFwo/ApSLS/\n",
|
||||
"iagPP4ndo3pzYOAY5Ie/27y+DJtGBT4b23qiEQJ/kNpfOQwhf94DRakJ9Rb4/YiItyAR8n7CKHso\n",
|
||||
"sBW8IOW1EEbw1qKZgQi16STJXsLqnP1UTvI9FRn0tJm4wO8bGrBv8AxgPz4HKU/gR5n/tbVofL6C\n",
|
||||
"z/SabqM22GMcFVqfL+PX9KQ4EFhL7U8sg1GBVxpEnQXeCxDRHYaI8T5CH34oYqtksQG4HrztSAR/\n",
|
||||
"JrAy7BnPXuQH1WYei0/ybSP4zbRH1kEPYAIH+6wDbxrhBCNlC3ytI/i5yPiAZiMtgn8ztbU2rMAP\n",
|
||||
"KiXXko5G8ErDqHcED6HA7yFaF2/99wy8veC92dxZjVwJLHMW2IecKPaZ+25rBAgjeNeiGQNs41Df\n",
|
||||
"DfRom0DtBL7WSdZWKgdwNQOVAu/TCxHGaTXc7kDkijBjMvdSGExzfi5KN6CIwE8E/gI8hVS4XJ2y\n",
|
||||
"3NeQ1gKPAydmrG8vEtFYi8ZG8AUEPsImpBLHbR62l6jAxy2aKVRaNJOAVbQN2EDvvWMIBX4X5bQM\n",
|
||||
"HoVEirWO4FtoTiFpBewYBIu9XWuB3020D1G5+PQ229EIXmkIRQS+DXgfcDRSEfAeKq2AC4HpiCd+\n",
|
||||
"JfDtjPW5Al9lBO/iHUa897jADyEawRuBDzyiAm+TnyLwB1rW0WfXEMJZpMpMsj5DmVGiz634HBV7\n",
|
||||
"tJkj+FVERdYKYj0E3u1DVDb26kQFXmkIRQR+PfCYub0LqWAZF1vmYuB6c/t+RGRHp6zPtWjcCL6I\n",
|
||||
"Bx9nOdEBUVbg9zv37QlkPLATvOeIWjRHAKvY37oKL/CojUWzmHIj+DORuWUF8ZD707wCv5JKgW+j\n",
|
||||
"fhF8rSpphiDf6fIF3uf4BrV3UJqIaj34yYj9Ep8VaTziiVvsIKQkSorgAXgtcEds3XEP3lo0bn/5\n",
|
||||
"uEWzkj0jVpj7UYH3ORu/U8PoRyEnoXIE3qcVOX7u8bVWUjMKfCsi8K7IDgeeoLajf2tv0chnvgoY\n",
|
||||
"YAa8lclfoCmrpqL4vAC/PU+llEw1At8C/BK4hlAEXeLRRFK5ow/XjIJ3vgCuH0bnPHjA2+xU0EBl\n",
|
||||
"ktX14LMEfhXbplqrJx7BXwu8sbr9imAjeGPRBEMhSJp2sCiTzf/xzmP2SqMZBT4tgn8a6I+fOEVj\n",
|
||||
"GZRv0fjMwo/kbQab9Zd7EpFjMpTKK+ly8fkMPkfUdBvwMeCCGm+j2ZgL+M5fhykq8L2BXwE/Bm5J\n",
|
||||
"eH4t7f3ZAYku1yYs58NXF8F3NsCbnqHzEXycvciJqEgEP9L48iLwOyetoq1/wKHeeyC4nwMD9iEH\n",
|
||||
"ehZwSif2KW7RfAB4byfWN9n8dwW+BThMcwq8nSKxp9OQzXYVXUbtbJoWyrdovgS8y7k/BBmct4Vy\n",
|
||||
"bRr7Wxtb4jqTuAyYU+NtjMNWmvmcj8+Pary9ZmAedRR4D/gB0iP9KynL3EoY5Z6OtCHYkLKstWj2\n",
|
||||
"0nkPPmndkFwm6czx6u0BDiE/cvHgYQN7hnusO2kUMIctM/sCLwRuQBqchfi8EZ9zcvfGZyByjNcQ\n",
|
||||
"JlkH0bkf+xRkoJdr0bQiFll9knl+an6lIwxCRHAbodAOR0RxKWUJvM9v8Lkfv/3kWguLZizwcuf+\n",
|
||||
"YOS3ULbA21moai3wo6htHgRE4G3Bw3S6g+3UhSgi8C8A3gCcCzxq/i4ArjJ/ALcj0dYSxNJ4d8b6\n",
|
||||
"OlEHn0tc4F2LZibRFsSbgTchrYE3g7ePfUMOsWbOMUDA5tkDzHKfBPrgMw6CKRC8D7gEeEWB/RmJ\n",
|
||||
"NC7bhTQvs2VznamomQzcS6VFswoYjE9PfMbiM6sT20jHZxqwypy8ymAQYoe5QisCv3fIszzxureX\n",
|
||||
"tJ05SHXXJ839Wgj8GOA0/PYrqVpF8Fbga2fRyNXUIGop8D79kZO69eBHO7eVEigi8Pea5U5AEqwn\n",
|
||||
"An9AhPxaZ7n3Imff44FHMtZXVh18ElbY3Sqa/qbPzTik6sayCfmxf7i9PcKBlv1snzId+B3rTuoL\n",
|
||||
"3IPPEuBhJIo/BrjCrCse1V+W0DVwFLARn4BwusEBpCVcJcKcmfMepwB/B8Y7VRQtZv07kB/M24DP\n",
|
||||
"56yno7wP6f44Pm/BgrQig40qI/h1J+0n8E7v9BZk4NRI4CYk4dmXqAffeYvGpwdikd2JlA1DNIIf\n",
|
||||
"gU8rfimjcychwVQtI3grtCLwPqfgc37J27AnKLutMTS7wPtMq/HI6KpoxEjWvYio1yOCtx78dGAF\n",
|
||||
"eG3OspuRRN6t7Y+0DdhDj4OLgX9y3/sPIF0tAR5CBN0OJhoHnGh+1JY3I2MEwGcKPp9GIpKN5vnt\n",
|
||||
"iLAPxAq8j4fPGHN7LBJlnprzHicjA87cwUGtSBRsk8ezgLNL/6L5DAdeR6VF1BnSI/j1x+9k8KoB\n",
|
||||
"qa8szhhgMz5tyIl9FOVH8MORE9UvkbJhkM95O/K5DEeCgx+WsK1JwD/pqMD73ImfW6E0Cjk2NoJ/\n",
|
||||
"E/CRDm0vnXHIb9VaNKORq9B4G/FkREy72hiDXyFuR5egUa0KIBLBBz2QH/r2Tq47zYN3E6yWXwJX\n",
|
||||
"R5qbPfL2B3j25B8gkf4UM5UfhBF8C96h4cgPawcysMtyDHC+sWGuBD4EvAMRFMzyQ5AI3lo0rwSe\n",
|
||||
"MBGmjf7zIjw7WGstYRTdithArsAPQK66yuQqJMn+MOVH8JUC//BVGxixqDcf7ndS5hp8jsTPTEZN\n",
|
||||
"IGxIt4HaCPwYs+7bgZeak+tgohbNacAJsUqbjjAJKVWuXuBlv86BioFycUYjn/MYI7inAmfgl1pS\n",
|
||||
"Oh6YTzSCh/iUmul8AvmNZSMngrGxx67E5/KC2ymGXFFPA3MVLgFcIzS2nUZF8BCN4AcDu2Ilj51Z\n",
|
||||
"d9yDTxB47/vgPRh56MkrXs6KF30B6Rk/2XnmYaSSZiADtgwiYCfwD2T6QFu2NhKpljkbeD3w/5CE\n",
|
||||
"mxvBD8aN4GXu2UHmNecAfyVL4KW7Yk9ElNyxBi08c8Ex7G/dY/ZjFnICKzuSeBHwc+TkUmYEn2zR\n",
|
||||
"bJnVl7s/BT0Ofif1h+IzCfgj8MGMvMB4wqqujVQKfBnVR2OA9fhsQj7rSYQRvBX40xHB72xlihX4\n",
|
||||
"qAfv82EzTiKLyYjFNilnuVFIfmo1IljHAfcg02mWxThkEOUoI46jkeNV1KaZRLGGdJ8Evhx77KXA\n",
|
||||
"ywpupyjDkat8a7O+G/h6yduoii4SwZdiz4CMfrQdKyG0aI4leY7XGN5hE9FLBB+yCujDyCdH0/os\n",
|
||||
"HO69Ackz2MjyaKTK6LfAZ4Bt+HwV+B7hzFTWohEP3mcGkq/4DFKOdg7wDeSHlIZE7+LpRyP4zUdO\n",
|
||||
"YecEz+zLAWTqw7n57zlOMB2Ci1OetIlqd9sdR37UUYtGHrNVNEN49K1wuHcvSI22fgZ8FUn+pwmn\n",
|
||||
"G8FvRITECvx6oKWEevsxZl0gJ/qZRJOsU5GKreuRwoWOIRH4OGQgWH+TqLSVTZ9EKr+ysMn3vPr2\n",
|
||||
"0cgVyVLgUqSI4ueE+YUyGIccq8OIMI4mGtHnMZFiAn8icKkUSrQzg/D3WxZWM+yV/RzKPSFWTVeJ\n",
|
||||
"4E0v+M7iBYTll3Zb5yER8m1VrGgjMBACuZQWQX2a8Q9MonUtHBi4lajAH4P44rchl7I/Nq+7Er/d\n",
|
||||
"c91BGMEPJuBK4EfATxDxmoDYH0PwU5N+k5GrCxDBsiLbwr7Bvdg1+iDSxuAZpJb2LGP/VIOtmori\n",
|
||||
"MwC5OlhF9khlu7yHz//L2X4/4CA+Bwitkv5AgM8eYAhBD9hw3J8gYdo7+cHORMp3/066cI4nKvA2\n",
|
||||
"gt+Fz2HkeM1OeW1RRhMK/DPIj9wmWTebfXsMiYLP6sR2xgBb8dmPzB1srQebAM1LSs8y+1Mkgt+I\n",
|
||||
"CPzlwAOI/XR+ibmd8chVwkbkeB1Avt/5Ai/7MBaYYWxR+/gl+M6JXuywScCNiHVqE+LTgan4DMSn\n",
|
||||
"F34pduYUJNCzAn+82cbE9JfUlkYKvFsHP4zO18C763ctmunAq8FLq8tPwAuQ0ZWTnQcXMnLhWFqf\n",
|
||||
"hb3Dn0MixpNMxHksIvCPAb9DRDuORPBT/jSYt5/eG0nKfgefxYho3meSgE8iJwwhGlnOJmyP7Nok\n",
|
||||
"rewb2oedEwDOABbhs9m8h/824hziMwyfeST3MulL5VSHIN7iMpOXSI/gw3Uei1wWZ1UFWXsGQotm\n",
|
||||
"OOHJXqysNadvJlmAXwr82exTlsC7A+/iHjzIVVZnBT4ewc8gatH0QhKj9wGnd0IkZWCesI7QpnkZ\n",
|
||||
"ctUmAu/zYvxEC2IW8Ceqi+CPAh7EZxUiyO9L+e6EyAn+jfjc3f5eff6V6KQ045DPZSPyfdlAmATP\n",
|
||||
"Q050cizckuC3EA1Qjkc64X4FuBJpGTEO+d49hVwxvwK32KLjTEGmGp1kfnOzkIrDhiVdG23R2Ai+\n",
|
||||
"LIsGogL/R+Bl4N3XgfUsJy7wQ5aJRfPcuD34bEB+0C9CBHk+PgE+F+MnjuKVJOvZnx7EU685wLfm\n",
|
||||
"n4TfLtZfJzwpzMfaNOIpL8bnChN1vA34hVkuatHsG9KXHZN6IqJi7ajLzbqejIn8pYgllFRHbac8\n",
|
||||
"jDMTES6IXj2E+MxFBNdDqm0gnsyL9vWx9gyEEby1Z8A2jlt48XMkC/B5SFkiiHCekeLVZ1k0IALf\n",
|
||||
"2clSkgTeTbIC3G88+nXYk7jPEfi8NbImOQGfmvJe4gI/1kSwL0FGPc4xr/s08DN8Xh17/WzkmOUJ\n",
|
||||
"vBvBA9h81WVID6g78bm8Pe/h0wefLzrC/y2kpHYGYcDyv8h32DKOMIK3Am+vsPKYiOQH5hO1aWYi\n",
|
||||
"yWzLScAj+DyFHK85ZplnCK/CXw1MpPMziE1FKszWId/NVcDv6ZBVWg6NtmjK9uDt+o3Ae8+Cd1cH\n",
|
||||
"17OCqMAvYvDqYQxZuZ9tU60F9Ankh2Qj+Cy2EzCZcQ/15OF3rGLTMWElhc8P8LnR3HuC8At7FXI5\n",
|
||||
"/QkkytiKCBm4IhvQwv7BfdlxhBXmRWa9T+NzmVmn+8N6LXI5fJRZ7sv4nGGe6wv0NRHYacZm8Qh/\n",
|
||||
"FCA/xBGRS2PhHCRaeQVycrnN2UY/fL6JXAXYKM5W0Nj3MwuxgawgDgWWs/IcDxgYsa5ExP4FOYlj\n",
|
||||
"TribkBxEnLhFMx7wjDUEtRH4WYQCvxXJDdkmfXcTjnj9d+RKbrKxCr6LXKXdCKzF50Wx7bgC/yxi\n",
|
||||
"U5yBHNf5yPflUrPc2cA3YvbDLLP9kWQ3QHMj+APIdwhzxfkCJPdxNfAFs/xM815ONrbI65Ay4zuQ\n",
|
||||
"kt1JiChfatbjEVo0m5Dv/HqqE/hVuAIvVwpTgWMISy1PRK62QYoYXoicdBYjAn8WIsaLCU+6j5A/\n",
|
||||
"HiUJO2PcM8CrkLkx/oJG8GV58O3r35e7VD7xROtCBq8ezJDl+9k805ZW/hTZ/x7IWTuL7cArWHn2\n",
|
||||
"YQ60SgIxmfnAC02k+5/ID2U5cB3wFfz2Jm5rsD1Jgp5D2N8K2yfZKP2Z6Cr5FPB+E2WNQC7jbyYU\n",
|
||||
"tdcgdgdYgReBuRH4b+QyNxR4mbN2I2FZm+UEpLXDd5GyzZ8RRvBfQiLpPxEOEnMj+CcRm+ZyohH8\n",
|
||||
"MoKeQ5HIyI3iTwI24Ue6mFbaNKGQuBbNFMLoHcoTeGsDLkMEdh8+beZ4nWksDoDvAO9yhPDnwIeB\n",
|
||||
"/0EEaho+s4GPY3vb+PTH53+B/yAcSGg9+H8lzDH9A/gi8BN8HgH+z6zb2n121PM6svMoNoJfAFzs\n",
|
||||
"nAzB5wA+3wc+iHw3ILTiXgNchFiOW4G/IaJ6DmKDDMJvP/m14bOL8iL4SWYdiwlLhCWCF+y+uBH8\n",
|
||||
"q5H8glSw+WaMi+Sy7Psdhs+/4XNmjjVlS5gXIyfwJ5DvbX98Po7fflVbN7pSBF8LD74zrCAq8Mvp\n",
|
||||
"v6Uvw5YE8l4CAAAgAElEQVT2YNNRPSDw8IMH2TT748DtjvCmsQOPVp552V7ChGsS9yEi+CTiez6O\n",
|
||||
"iOwyQnsGRAT74dPC4Z6DONAKOye0IhUJSyNr9HkQ+aJ9GPEo70AGbx2F1AePI7ys7cvwRYOQpN0J\n",
|
||||
"iMhfgkQ97okjqVTyRORkshBJND9FKPDnm/dxD6HAhxG8HL9vIT2NogIv34/QJ/d5CXLCuD22/Xuo\n",
|
||||
"jJZGALvx2793GxEhcAV+CXBEe0TrM8pYbcfgJonlBOnjcxuSQHetrDCCFzFchXzO9rX/dG4/gVxl\n",
|
||||
"3YCIwNXImIi3A2/Ab3//vwJeYrbzDvPe/gWfm8zz65DP5kKkkgjE558M7VeE30Ui6COx/ZjkWK8k\n",
|
||||
"LdEqkfBw5AR6CN9cJVUiJ10RvRmI9fMa82e/q39FriTmIlcOvzX7bO0ZkM9kLPEI3ueiCpvKb6+R\n",
|
||||
"TxJ4K9z3Iy0j+pr3PN88fy8SAMxGRNg+/gvkcziOMEA42WzvUuT7cRZSAXUvSUUQcswmIrqxGAle\n",
|
||||
"njDH2o7+/ia++c34fNp8JjWlUQJ/CClprJUHvz93qXxW4wqYz0F2TjzA4NUtbD6yLyIcJ/HNp9fh\n",
|
||||
"868F1redgICFl+wgLJmsRCK+a5Af6ZvNYw8iCd02Z7kAEZGJQCsHBuxh6/QhSFI16f3L5bP4oD9G\n",
|
||||
"IrOjzGNPId6tB/RlzjdGAzcglSz2B+l68BBPtPrt/vkS5Grg84iIzUD61ww023yYsPrITbKC5CF2\n",
|
||||
"ExX4pcj3YyFwJD7nIVczPwY+GnuPdyEC6CYwXXsGxA7ogSvwcrxWmX29DhGJ9yIJ8++aZfojkd5J\n",
|
||||
"Zj/fhox1sFPzDXb2G3OssgbufRWxsn5gIt2rgcvx220e8NmInOhfjNRUfxg/YgU+i1x5XGnWAWIJ\n",
|
||||
"3I3YA+Cz22zrS8gIW5ufWUmSDy+e+jBgh7nyyGIT0oxwBPL9+DXy+3sZYdfZlchv/TLkBPxbxCJ8\n",
|
||||
"KeFVlR0MGEbwcgX7O9xWwlIKuhax+KzALwOGm+9fVODlqmFJ+8ldjuc65HguNo9/ATmR2hPFWUjg\n",
|
||||
"YDvIvhN4Jz6vRU4WUk1UOVhtHFLdtI/wd2JtrZvx+SByBXOpOUm9Hyrst9JplEWzx1Sq1MKD30U0\n",
|
||||
"OusozxJPQm6eeZDAC9g2dSBh5n5G/IUpLGHXmF+wa9xzZAm8xWcTvnNVk3yFsBqYhBe00DZgNYf6\n",
|
||||
"DccPknvQ+DyJz0VIzffvELE9EjiFBa96joN9DgFT6LNzAMfePASJpkGuKCYh5YvrnTWuIRrBnwA8\n",
|
||||
"js9hfPaYyG8PchzfDswz70ESW9EaeLuPexB74UkIeiInBTudn7Vo3gP4+HzXicrt69cgP2C3vjna\n",
|
||||
"ulrEfDuV35GnETvjRKTPz0sRkXilEfeLkO/oJfj8BBHcd5sIfxQ22g15BjeCr+Q2s73fmP26CZ+/\n",
|
||||
"Jiz3W7NcGxKBuvwTeAc+v3fe3wKkgsb9vnwNOX6XIjYFJAm8zwuR79Qs0rvBussHZr2zCMX1p8Dd\n",
|
||||
"7SccWeaviI+/ADkBLUM8arvfdjCgW0VzPvJZvt/Z4uVI6/KXYwVejvlfEB/dFfgXIVdI/xXb678h\n",
|
||||
"0fRSs38fNFdMVuBfiBQ9HIcMGjuTMM9zGAmUliD2mYu1ZzDHZBtyjF1+jVypXYF8nlljXkqhV/4i\n",
|
||||
"pbOX6IjTfsiHVpZFcxVhRNAZ1gGjRGjMCNvNR8GUeds43HsEoR88vdDafFbDus8hg5+yLJpqWAVM\n",
|
||||
"wjs8kAODliAJSttLPW0/DptbG5DP/wIWXTSMMY8uZ9iyOZz96WNZd0IbN/55iVn+oBGQY2KiES+V\n",
|
||||
"PIEwmeWyABH4j5r1bTBCPploktXu3yfMrWGE/XWswJ+BBCVXpL4/sQnOI6z6iEfwIIKSJPA2Sbjb\n",
|
||||
"7MtGxMc+H/HKb2w/Bj4PI9VSL0c+h/Wx9S0m67shwvShjPdh+S3wOeRkEj3J+2wHvp+7BvG53xd7\n",
|
||||
"dBWVA8OuQYT4U4Sim8ci5LdgxfVeKier/zPQw3z3DkBFZY/d1noTHLSZZT4IfBxJ9t+P5Bq+gZxs\n",
|
||||
"bQQPEhlfguTybkc+y97Ah/ArbLy/Ia0k4sHBZvO5H4mcMFYjdfNP4EestgCfzwG34vMfzmcSCrzP\n",
|
||||
"MnyOSgjK7kROOuORQObFJCFBw3FAL3weSFymII2I4HcTzppUgwjeWwvegfzlctfThuxT2Pt849E9\n",
|
||||
"ONxzPWJFzEIun4tG8CCjWHfTHsEHMyCIfwGrYRVwBN6hfuxv3U7xBJWNrBYAc1h5ziA2z14B/Asn\n",
|
||||
"X3sqf/50PIdxPVLP67KGqIC51QouC5BL+HnOY48g1lDconGxNeS2fHIJ8gO+qV2Ak7mTMGEMEs0/\n",
|
||||
"HVsmSeB/AbwZv92XdR+3/vevY899HRGhy6gU+D9ACZNX+DyDnBx/3Ol1RYlG8FLlci5iiZxFkQhe\n",
|
||||
"WIicKAYCzyIWYzxY+xHZs6K5Eby9fzYi1l8CvorPucgV9YfN9kYQFjbchpzUj0ZyDIeRWvTvJWzr\n",
|
||||
"d9ikcyXzkbzXfiRH9Z+QmH+wNtkxzmMvJQwqiFht4WN7ERtxIFKXfyw+PfBpwbb3lqq3DUgO5YyK\n",
|
||||
"dVRJIwR+CeGou1p48GXiRKmBx+IL+7J71OcIG3rdRnUCPxBJLtu+NKfSuYZgq4HZBD3bONzbViNU\n",
|
||||
"0251AQGb2H7ECFaevRp4G2tPW8+zc6JXdjJYJd5J8I9I46zXI4mwkwknZ3d5ChE/t1XEw0jkewGV\n",
|
||||
"wmiJCrwkLu8gtI7S+CvS6dMOEHsxEkG6bCAu8BKRJw1Q+w3yfb0rEskJv0I82clIVZK7viX4/Dxn\n",
|
||||
"X4vh80l8x8oqh2eAc/BZhyRRvwlcj8+jiM2S9rnEWYRE1ItTbESMbdeW+JywCThIVOAfRgbrfQs5\n",
|
||||
"ad+FVAbtRKLw9e2WmN/+/RqFtUXkKjFpX7bhp54sH0Kid7CN1pIEXt7nrdiuoVJ5cwESCOXxZSRP\n",
|
||||
"tgX5fk9GbKT5+CxAclcvxedo/PbEeYdpgEXjBYRJiFp48GWyltCH78vu0Yf42tIbEJvleOTS+T0y\n",
|
||||
"9Z+XV0UD4eAa21nyKGBEFa+Pswo4ikN99iNXRbZhU1EW0DZgGvQ4lycv38BLPvgMf/jaEoo0EpNL\n",
|
||||
"2pcjCb3PIMfqqYQl70TK4cz7C17HwX6P0Gv/R5CywOtStjAEse3MnKmBh+9dVGC/9uAzD0lm3YNc\n",
|
||||
"JcTHKGyE3KZcdn3r8fk1JESDctL5t0Lr6Wr4LMdnMPLbOwuxOKygvAcqxjiksRCpgInnB6rZlwNI\n",
|
||||
"R1C30ukx89xB4H/w+S2hp30blRbnrUir4bzEcBYfc24/hGjSwynL3oqMgfkUkoi92Vhm2fj8HSnn\n",
|
||||
"hbBy53LkiqUfYgmVpoWN8OBd7CTZgLc3c8nG8Cyhz2zE2Qsg2IyI4INIsmQUxS5pBxBG8EOQS8re\n",
|
||||
"iGfZkQhNKj8O9t1uXr+X6iL4G/jrR1YD57LjCIDZbJn5O9mnoIc0X8tAErdzkRmvkqJ3G12Z6Dbw\n",
|
||||
"gOv4/PrZfGjoVKI17HGGAtvlexEcIrS3inAd4jn3QhJ+8fexkequXl+dGp02MyKoa5HxCj9zHq+m\n",
|
||||
"bfdSpCouPvai2n1Z4tz7LGHPJfu8K7TXEw74s9xA8bxB2j64JwcZUxFNnLv8FZiJz2eAt9Kx0apP\n",
|
||||
"IJVyHjLKufTvWKMFfj/iZ+cNEmoUbiKxhTB3sBkR+/0Q2GHpRQTeRvDWohlBOFl2hsAHps+0tyT2\n",
|
||||
"xGqgF20D2sy+7acagffZAh+yX+p+5gtma7v7UGQ8gVxWFmUI0Jf9Q4aknhCiy1qhsT58UYG/DZme\n",
|
||||
"70pkPuE4d1BNi+DuKO5lIdG3Hb1Z1jqzW4tIDuax2GNrKJJwLr4Ph5Grk7Tn25A2EGcglks8z1OE\n",
|
||||
"+UgO51O1+o41WuCtgHRFewZE4G0XQ1fgtxCKja2WKHKJaqPQHYiVMh6xNUYQnU4wznTgIQiGgxdG\n",
|
||||
"GT578dlM28CDZt+2IcnOarAWlJ27tq/zv4wBYy525Gts0ojgPOCu2BVDksBnRfwh8uO7ASmxqxw9\n",
|
||||
"6A46Usrga0iN+/MLnz8hgxI7yhPm/82ZS3WCRgu8HZDTlQXeCqDbnMp2agRJGs+AYJQ872VFmW6S\n",
|
||||
"dRpS3bGO/GhyMOIlz6Hy0nQVBwYOR64Aqk2yQthZz3aQtALfj+w67o5gBd4ZCRh4SLLyGKKX5a7A\n",
|
||||
"Gx++Kr6HlHYuy11S6Rw+32j0LjQpC4Gr8RNzV6XQ0OmkCAW+rBr4snE9eDeC/wdh2d9ipARsFdGG\n",
|
||||
"Xkm4ZZIgJYR2vk4gOBeCb0LFYCVbV/yShHWuYv+gwOxbRwR+PDLwxAp8XOjLxPYud8XazlMbn43J\n",
|
||||
"JllBAoDqJsb2WYxf6uQUilIuMsakpjM+NVjgvQDaJ3voirgevEwQAYD3JfBsc6d7kEusH5IfidsI\n",
|
||||
"fjeSmLICb193PZIofX3sdXbO1RSBHwwdF/hxSKIsLuy1EPgki8ZOhhAfHGOSrEB586YqyvOKRkfw\n",
|
||||
"ID5vVxX4bUAfCAYiApRgv3hrwfsAYrfk9ZM2EbwXIPbHU7QLfNAbEUAfGG6sC0sLUo54PASDIJjs\n",
|
||||
"PL+AnRMOIxbNallHUM30c/EIvi9ykqmVwO8gGo1bgU+K4F2BH16D/SmJYC4E8aHritJwigj8dUiF\n",
|
||||
"SHyEn2Uu8qN91PzFB8TksZ8uK/BeQNiTxrVoksjvLxNG8CADSZ5AErYjzDY2gLcLKb10I9pWZDDI\n",
|
||||
"/cBNSMQto9x8ruX339og++btNcvMLfT2hKQI3vXky2QMctWSFMEbgQ+Og+A0ogJfVmuHWjGLsEOm\n",
|
||||
"onQZigj8DwlHnqZxD1K9cSIy8W817KPrevAQ2jRukjWJ7eT7xG4t9xzwnib04N2JHLYQjVhtnfzP\n",
|
||||
"kYkj/ka0D4xbR38nyVZOAkF/s09riQr8DmrnwecIPO9AapBPonkEfhD5J3dFqTtFBP5v5Atw9vyM\n",
|
||||
"2XThCB6QCH4C+RH8NqqK4D23ImcE0hfEVuYkCfwu8L4L3sXIyMyxlc8Dlb1YshiHVPHspTKCr5VF\n",
|
||||
"s4DoiXAScsViBb4F6b3yfcLS0U4IfHABBK/p2GsLowKvdEnK8OADpKXm40hzoKOyF6+gK3vwIAM4\n",
|
||||
"ZlKORZM0GtMKvBvBu4lXCJOslvVEZ1Nyn38CGArBqRA8KP9TsZMu7CNaB19rgY9H8IsJBb5V7ntX\n",
|
||||
"G7sKOhfBv4xwLEMHCb4KwSkZCxQU+MAzuRZFqQtl1ME/gvxI9yANd26B1PkMfef2PPP3FNmDfBrN\n",
|
||||
"U0iviLWEM9AkUcSicT14i/XgJxEOfEiK4N1jtI7o1HSOReMdhuAupGnSDqS/9YMkcwLyvkzTt6AH\n",
|
||||
"0BM5WeQIfNAfGAfe0oTnPgI8Ct7vncd6IyK4mEqBf5Qw59BK5ajezgj8dAo1zgouRN5P0mjIU5AT\n",
|
||||
"00MpLx5UcP9eDrwBmfFIUdKYS0kTdZch8O6P8Q9I97e05mF+5UPe5SXsQy15EukZs4PCEXzQCpwH\n",
|
||||
"3i9jyyRF8FbMj0BambqPWZIieGPRBL2obCvwA7Pfm4nPUdpO8E5kCr1LCLt69kUsMyv4HtKPvBW4\n",
|
||||
"BTy3u93lSJ/3pPVfiAi2I/CMNvuzifYTYdADySX8lGgEX6bAzyD7c7O8CznuSQI/mOj0jXEGAQPl\n",
|
||||
"JOZldU08nerLWGtEMB74EXgF8zVKHZlHtL32x5IXy6cMi2Y0oQc/x9zuypZLtSxGousRZAvFLkQU\n",
|
||||
"eyOi9/OEy/qECN7bh4jqMaRbNPFmZOsILRrrzzu9LLy7wfsMYpsdTwXBSKQT5lngPUClwNsunwOR\n",
|
||||
"hO0NVA7iOgU4GYI+sXV7yAlxdmz5MWa/nwMGmOM0knBSD1fg48e5gwIf9CGcWCRruUFI3iJei28Z\n",
|
||||
"RL7AQ/4+npCxjXozGTjHzJyldFOKCPzNyPD4WUid9VuRWZOuMs9fhpRQPoY0se/qEXmVeG1IO4KT\n",
|
||||
"yayi8QLCKH4s2F7WkR9QWkfELUgiN62KJsuDz+pE+SQwK8H3nQk87dgrNskaF/gWxI//CTDd2DKW\n",
|
||||
"U5CTebyf/Xizv0kCv94cp21IFG9n5dlNNMlaVgQ/GfmO54nqhciAu7QTQZEI3i6XxYkF9qVejEA6\n",
|
||||
"mY7PW1BpXooI/BVIMq4P8oO8DrjW/IFMFHAM8kM/E7plI6enENHOu9S3Aj8GmcWmjajfmlZquRnY\n",
|
||||
"Ad4O535SmaRlIzI4qieZyV9vD+Ecmy4ziE6gnWTRWIHfJV0zWUj7HJJBb+Qz/wViO7gcjQQEk2W5\n",
|
||||
"wINgAO0CD4StByZRKfBJFs1OYFBs8FcCwSwI3Fl2bJfPPFF9JfDL5OWCHmaf8gR+H5mJ1mAMYs8U\n",
|
||||
"60Nfe+wV4tSG7oVSU7rCSNZmwDYDymtXayPTMUhC9k5kjkfCKDrRo91MGL1DmHi1xETcO4iI5EiS\n",
|
||||
"LQ2Xx6mc3Hc62QK/39x3xVYmyxaOQko6/0TltGLHmGXXIA3VXo4kcl9NKPC2eVhSBJ8g8F6b2cc8\n",
|
||||
"ob4GmYDBfZ9uAjeBoB8y3dtNKcu1ILbaQAiyLJxVZFfS2Dlru1IED/IZNSlBa6VFqLiowBfDzghU\n",
|
||||
"NIIfi4iZa7VkDZRKEvi4RROPaq1NkzdZyBNU+vAzIDLBghX0flRaNPY9P0w4WvMUc/8ftEfwgY0E\n",
|
||||
"j0ZOiAuRK4eLkNlvjias87e9ZY5A3vdu2VbQB/lOJs2pW8SmOQV4qWktYd9nnqhORI7/cpKj60HI\n",
|
||||
"57qC9Ci+iMCfSPuE1HlXInXBzkHQzBH855EJM5QUVOCLYSP4aiyadUR7qNjZnJLYQih+kJxkjW97\n",
|
||||
"HXIiyavPT4rgYxaNdwiZE7OVqMCnRfAnIwK/GBgMwReApRCcRSjwi5Crl/OA/0OiaTs9n73SOROZ\n",
|
||||
"09RG8GZ7idMX5gh80Ae5enjUbNO+z8fIFviRSGXPrpTlBiMW0XISBT7oSTgaOOsEdAJSZrmfcMxB\n",
|
||||
"DsGZMjF7TRhh9qeZBX4EkmdRUlCBL8ZSRLx35iznWjTVRPB3EJZIEn1d4JEdwSc95/IEcEIYNQYe\n",
|
||||
"lR48hB5yggffvp7ZEPRFIuWHzAQd/0RaWXwU+F/EvrER/CVIInahVAu121NbEbE82rx+F3J8sq5G\n",
|
||||
"8iL4o5GmaTcBl5rHpiMnuH4Z1SKjkJxGmsAPMttOEfj2Y5Qykjn4AAQ3IPOePpaxnSTegVwBdZJg\n",
|
||||
"HAQ/jD04Ajm5NrPADyKcr0FJQAW+EN4h4Ajw8gQ+yaKxg3qSBjnZ9f8BvDucB3YDvUzVSh8gAC9u\n",
|
||||
"WxSN4Fchn7Pt+TIa2A9evP3EPkRAUyJ4by9yovsRYWQMMi3eC5B5NCfL8t42ROBPB/6YEJFvRcT/\n",
|
||||
"AVMmGovgE8kTeHPS4RbgZcYymoAI8x4kyk7CRvB75D0H8bEhORE8g8zzaSOZX4mI/5+QY1KNwA+m\n",
|
||||
"nKTsNOBFscdGII3pml3gtQooAxX4wuSKO8iPfAIyGnQn0Qg+bfBX0rYCwkqaNAG3EfzRZI7U9AKk\n",
|
||||
"quVM80BS9A6VAt8vYdvXI8nj48I2At4aqf7x2pAo3s4Yb+ez/GPCtrYhNs/d5n5Rgc9qg2yvKtYi\n",
|
||||
"A6z+CvzDnBh3kS6UI4GN5ji5yV6LFfA0gbcngASBDzwkD/EJ8N5kkuPPZexLnEFVLJvFMCqP3Qhk\n",
|
||||
"dO6AKttLOwSvgaDDg3BKYDAawWfS6Cn7uhvbgXNpr/cOXIG3UX1R7Gt3kCx66xEr4ngqPfY4/0Cq\n",
|
||||
"XX5KZQWNZS+ZETyA94XszXg/geCn5s5mRNyT5qy0J7okgU+7GikSwRsbwntj7LmsqHkUYYLbLudO\n",
|
||||
"VZhn0bgRfHz/hiO9mrYU3Jc4ZUXwRuADz7maGoFcuSxDovi8SdCTmE1jrwDMCGIlDY3gy2Ubklhc\n",
|
||||
"Z+4/h/i/fQgTr0XJi+DXIf0qfiZRdCbVRPD7SPbgC2InzvYC8M4Hb3vCQtvMem1vF2uhDCLXogl6\n",
|
||||
"Q/DC6FNBX+S4P57y2ixRNRE8kBxd2wh9GTAloQLGCvwOKi2amcAzMYuqGoHvQAQfeGZ+YJdhRAZ8\n",
|
||||
"Bb3NercTCnxHGEVjBdYcn0BFPgUV+HLZjniCJlL3AsKSQHegTxFsLXyabbEOqXz5vwLrehg4ygw4\n",
|
||||
"OopoiaQlx4MvlfnA58Okq3fIbHNUxvZsBH8m0jrBZRawwuQJkniO7Ah+k7mdJL4mgvd2IEI+KeH5\n",
|
||||
"NA9+FtKNNL4vRUV7cMZ+p3ECMv7CxeaBBjn3t5qT8TLgwxA86ZS6FmUkNRf4oC8E8WOOU720CrVp\n",
|
||||
"UlGBLxcbrbpCbq2WsVQXwdvXpUXRS4BTwVuZ8FwMby9Sy/8RxMq4K2GhIh58SXgrwItPcbcLOQnm\n",
|
||||
"CfxMYFwskp5ItMw0TjURfHw5G8GDHMNjYs9nWTQmgi+8L3E6YtGMojLxaLuc2v0bgVwhgpwsb0Rs\n",
|
||||
"snh7iTxGkp68roLAg+BbZtRwnIuB7yU8bu28NWiiNRUV+HKxlSmukFuhrjaCt7XwKVG0F4BXjW96\n",
|
||||
"H/BfwOvB25zwfD0j+CR2IxU+RQS+D9FxAhOQH3oaeR68G8HHBdV68JAv8CkWTeF9cQj6EI4mroZh\n",
|
||||
"yJy+vWKP2X2FsHII8B4F7ytIwtWdRAYIjs/pX19WBN8H6eaZNO/uJKJzH1jscV9LagQfTIfgyyXs\n",
|
||||
"X9OiAl8uSRG8HexUbZJ1BVLeVlYUfT3wZvDuSXk+rw6+1uwmu9+PK/Agom6xLQ/ScEQ1OD4Uv8Aj\n",
|
||||
"TDbGlmunSAS/g3QPflHssaIWjRVjs2zQD4I5BV43FBl7MNJ5bBhwmOQI3mLLbl1uoLLXkEtJEXz7\n",
|
||||
"wK+kSHwicuKPY4/7symvAziWUsYR1IugT9mjnFXgy8UKfDyCtx58NRaNFZO8VgQF8R4D78aMBZIi\n",
|
||||
"+HoLfFGLZjXRH3XBCD6YiozIfRaC1yKCvNc0U4Nki8ZGiiC5g2NTnt+JJPzMbyroiVQsxfMdRS2a\n",
|
||||
"QUQ7XJ4FfKPA66wd44riMOT42JNGksDbyeVdxpJamhr0MOspI4K3J4k0gR+ZMFCtQATPaGBsF2kN\n",
|
||||
"UYSfAi8uc4Uq8KXi2QqUuAc/DvmhVtMnfwFSGTKY+ohsgTLJmlJE4IchpYr3EBWDohH8eGTk7NVI\n",
|
||||
"22vHqmhfLqmKxlo0TyPtl137wwiNdyj2etPjxouPXi4q8IMR0bXrG0axlsnWjhkTe2w5VUXwQW+z\n",
|
||||
"XFqN/BBEP8oQ+LwIvgeV9o0V+KwIfjRh+W0zMBrpz1QaKvDl8xjRhN8WpHJlQ1hCWARvJyI+x1Ef\n",
|
||||
"kXUtmhonWRMpIvCzkZPnUqqP4FsJ56C9B0k2jyZMsNrlMiJ4bzcSMU5Pfj7iw88iuRy1qEVjBX6g\n",
|
||||
"iUCHUWxi76FI7X08gl9OdgQft2hGIlZPmsCPMvtXhkWTFcFPQD77uE1jrbO8CJ6M5w3BCAjeX2RH\n",
|
||||
"a0wrJc/4pQJfOt4ZsSTmFnJHm6byJDJAqR4iu4+w2Zj14OsZwe9CJqDI8uB7I0nLNbR78IFH8SSr\n",
|
||||
"qWTy1iG196cRjeDTqmjcgU9PErVpXIF3ffiTSB48VI1FsxX5PAaSGMEHM2RC8AhWzI24Bb3M9lZT\n",
|
||||
"nQdvrwDSBH4kUqLYL6X6pRpsBB8T4sAm0x+nUviKRvBQmVuIcxLhBEaNZBAq8E3HVsLIs1rmI5ds\n",
|
||||
"9RJ4j8Z68JAdwYMI/FrCH/VQoA28rBORFVUbwYNMRH4hlRF8UhWN26ZiPtFEa1oEfxrJk99UY9HY\n",
|
||||
"UcwtyPvsa/rXW14NvCUmsEORNhFW3IaY9WwjP4If4/jVRQR+I2LtFeyOmUqaRTMOmbDlWSojeFfg\n",
|
||||
"42WzljFkR/iWiUQngm8UrSQnlDuMCnzt2YJEntUkWC22D329LBoIBb4FaWWxL/UV5ZIj8F4bIiZx\n",
|
||||
"gc+L3u06nQgeEIF/IZUevCO+QW/EqnJ99L8Ab4TALT10BX64EZvTkWZeSftStIpmp7O83Z5r07zM\n",
|
||||
"7K87acdQJFdghcL2QNpJNIJ32yfgNH2zXndRgTeToXSKAWZdcYG3uZWNJAv8DjNr2R6SSyxHI+2j\n",
|
||||
"8yL4icCQxs5PG1g7TCP4JsP+kDpq0UD9IngIBX4gFZN515S8CB4kEo1ZNLkJVohaNG4E34tsi6YV\n",
|
||||
"SaDGJjTn18CPTeTsCvzfkcm7JyE+uDuJS3xf8nAjeFfgjUgHwxGr6A5kMhHLMKIRvBV4t1mbeyXj\n",
|
||||
"4to0YxBhzRL4TSQ3aKuW/ki1UZrAbyDdg4f0KN0KfJEI3qNYjqNW9EW+jyrwTUZnBH4RcIj6RvD7\n",
|
||||
"EIGnTtu1FBH4OxFfexvQx/QgKRLBuxaNjeBtH5wsiyZuz1g+iIjaN4kKzc+AVyFXBvennBw7K/BW\n",
|
||||
"hM4D5iGN5E5yXpcTwbdXxyR9H+MC/wzZXTitwHc20WpbDrTGLKgJpAu8+9m4V3SGYABy5byIfIG3\n",
|
||||
"wULSVUC9aEWCAhX4JsOWRnbAovH2IZNnJI08LRsngm+f4ale/juEAp+xTe9N4JlOne3JNSsCWcSS\n",
|
||||
"rGCaoC0m06KpSLDa/WhD5po9lkiDNG+FWecHSbZnoOMWzVBE6KzAX4i0RXZm2gr6IqK2lNBiiUfw\n",
|
||||
"Y5H2yAcTtukK/FhEHPMi+DIsmv5mPfFErxvBpyVZITnROho5iT1LMYtmF3Xx4YNzIXh7whODkEBl\n",
|
||||
"eJlWkQp8zfH2IV/ejkTwAGeC91T+Yp3GNura7/yvZwS/Czmp7M9b0LAWEfeJFIvgRyJC4o5FeC1S\n",
|
||||
"MmmJWzRpETymjPV84CpzQrTcTDhTVdq+dDSCd2vZz0FaMT8KnGg83KFIHmCz3A56UenBZ13xuGJo\n",
|
||||
"I3gj8MEoCC6G4DizrTIjeCvwcaHOsmjiEXw8Sh9NmKDNiOADz2znCeoTwZ8PvCLhcTtOZiclnmhU\n",
|
||||
"4OvDU0jrgQ5QMVCmVrgevP1f7wg+bT7WJGyTqaIevOnyGfHTH3VGsdrlbGuA/qRG8O2v3wXed2MP\n",
|
||||
"/hwR5YcSXgAiZP0LlBZaAbP7NAzp/GiTgWMQW2MdYuNNQAR+q4nOtyFWTDyCzzpeSRaNjeDfAnwV\n",
|
||||
"6QMH1GAAAA+pSURBVPH/r4Q9fMpKsu6l0mqxJ++kJKv72SRF8GOICHzqaNbBiDWynPpU0swmOo7C\n",
|
||||
"Yq8Ek65WOkwRgb/ObHR+xjJfQy5NHyea8FEA8OaY2uuuTFzg91F/D76a7a0GvoWULOZd4dgTVVJi\n",
|
||||
"Mb5cCwTjkUj4E+TPwxvD2wCMTS/b9A5TTBTdCH4EMkvYOsSiMRPBeG3mhGVtmmGEDe9s1GtEv73V\n",
|
||||
"Q1YEv44w2o0L/Cgk5/A64MOEg8TKSrLuoVLgXQ9+VEykC0bw3nOIgKfZYvaEZ9t6FyC4AoK3FFu2\n",
|
||||
"glnIvALxyZbsmJON1Fngf4hcVqRxIXJGmoHMz/ntEvZLqT9dIYKvZnufRwaBjQUvT7jtVVDeSdZa\n",
|
||||
"NGcj1s1vSZ5yMIfcq64iNo0r8JMQAbItieOdSY1N0y7mEAq8FX0z5yxTyBb4sRC0ICeUtUQFfhOS\n",
|
||||
"2N1s1r2Z8iyavUQi8WCoeXyD0wLErXIp4sFvcJ5Ps2nsCc82BTQEvSE4LeU1F5HdhC2FoDcyb/FW\n",
|
||||
"wjmSLaZiq/4C/zfCqCCJi5FOhSCJpSGUXKyv1IUkga9nBL+a7KvEGN4W8J4s1v7BO4wIUd6J4ADy\n",
|
||||
"m3gxcAd4Png/Kr5PhSki8G6S9QhCgbe/L1fgbQQ/lPC3up5Q4LeaSH8n0t8oTeBXIcHaBPP655BW\n",
|
||||
"CT2Izl37SeQKYj/lWjRrCIXPjARu/3zj1kWRCN4eI8d6ClpMozmLjeBtU0DLFUhVVBInEu3WWZRp\n",
|
||||
"yHt8ikqbxlo0SXZUhynDgx9P1NNza5SV5qHBEby3CLzLa7iBXeRG8F5gljsfCWxqRZFKGjeCPwIR\n",
|
||||
"bivwSRF83KJZB1xCeHIAEcSjSRV4bwVykv0veX27ndSCiKstKb0DuXqCci2aRwDbEvlkc9/iCF/7\n",
|
||||
"bE67nOeGx3rXp0XwZwA3Q3CKue9aNG6S9d3ymniuJBiA2CwjqJ5ZyBiFJVQKfE0smrIm3Y4nMNIS\n",
|
||||
"Zb5ze575U7oGSR58PS2aWlNA4AH5kQ2nY5NQV7Mvtj99T2CE8e4NgUf4g7cWzRJCH90mEC3Lzfpm\n",
|
||||
"EpbUfgnxys8lFPQdiOBnJaU/hgzY+rW5v5NwhKWdJCRAau0hnE8XCMZIGWvV2Ah+EdLbZjIi8Lc5\n",
|
||||
"y2wA3mBE/GFkEJ6d//cgBBuR42Lfm3uMXAtnPCLmX4PgLETg/0okgg9OQk4IuxAhd8dKHGv21RH4\n",
|
||||
"YLhcUeYyGxH4DSRH8DvluZsuIqqVHaaMCH4tUT9pgnksCd/5m1fCtpXyaHSZZK3ZRb5FY5f7Rzhf\n",
|
||||
"bM32xUbwHwEWQXCy8/xAZDxCG+02ScSiiUXwXoBE8S+mPYL31oH3XvCGg2dH1O5Egq+ME513HxKh\n",
|
||||
"2/Xb8sp4a2WLieCDEcDiYo3HKipaTATvBYjYnoNckTzsLPNl5Dt6A3Aplclv17/3iAq86yqMA36A\n",
|
||||
"BLefJZxfwI3g3wV8h8p5B0DsmXm0WzTBVNIrpuJYgc+J4F9/iKhWdpgyBP5W4I3m9unIl3BD+uJK\n",
|
||||
"F6XRSdZa80XggQLL7aK29gy0V8YERyC96T8K/B4COyeqWwJoT7JZHjyInTGL7HzZDnld7snrSuQK\n",
|
||||
"AERIJyAnnKRJzW2SdRzhYLIMgnOBpbEqEptkBUluX4wItDMblncveNcAHwA+TaXAmwqcoAfyWe8l\n",
|
||||
"LE1eTRiEjkME/zJz+3TkCshU0QSe2f5NJI6Q5URk/MFQc/U1ieKTisw27ynPoqmrB38zMp/nLORA\n",
|
||||
"vRVprWnba96O1OcuAa5FvCul+Wh0krXGeDeClyV+lseR73Qt+SUiQrcBXwfv68B3EWGFaALRnmS3\n",
|
||||
"EbVo4gL/qPmfNanMTvIHhQHeavCWOq+ZTnL0DmGS1Y6cnZa8WOBBcCqSuBxJ9KrfWjQgAn8p8ERs\n",
|
||||
"AJnlp8ixSIrgxyFR+anA2c4Yh9VEI/hn5arGez0wCrwlhBbNWEQXV5Es8CcgfYx2IkntcYSdVzMI\n",
|
||||
"PEIPfhkwNXa1Yz/zunvwVxRY5r2d3RGl4ViBP2D+/4RiEW83w3tHHbbxCwgeRKyAz5kHb0GO+b+T\n",
|
||||
"H8EnCbxNSOZF8AUEPsJORLQ3pjxvk6w2cp+G2CwOwRnAL5DJtd8NvMcst9wsYJOsIDOZbSVqzzh4\n",
|
||||
"hyD4b+CVsSfWIhbVmcCRsZO5W50Ta7TWPnfDDkSkT0GqdwIIYgIf9ELGXTyB5DpGOM+PIDsguhD5\n",
|
||||
"jW02695KtEDFRvAbZF+De4DbwPt8xjpzKSvJqjQ/e4A7w5Ge3s0N3Ztuj7cCqVaxPIaMUp2GJBhX\n",
|
||||
"mMddgd+FiOF4Km3QRUgUnCXwOwlP4EWxEXyawNsk6xhkYu+kCP4VyHia/zHidr5Z55/M804E7x2G\n",
|
||||
"4I/Avem75P0WGaPgshYp3/z3hITnesR+6UtqJ03vMATbkaS0TbCvJVrvPlXW5T0HwSbkSsRW54wg\n",
|
||||
"PGEZgqORmvmdwMeBi52R1EsI/X8Ip37cCcGxyPHcTidRgVcM3kGkQ6HSELzDEPwB8X+vIbxydgTe\n",
|
||||
"OwyBLbGM93M/BMGrifjWFfyCyoq3PHYik5fcl/K8jeDHIJHt1IRlTgE+FxM390TgRvCA94Yq9xFk\n",
|
||||
"1O3TyGjbGN4hCNYjNs1o0pPMW5GrgM+a+0mtE2zC2kbwrsDH+TRi9wTAq8D7h/PcY0gi+c/mvjN7\n",
|
||||
"mreI7M+xMCrwitJ1uB34PvCwIwauBw8S1e1N9qe932ev3ita7eGyExHtW1Ket0nWMUh55anRp4Me\n",
|
||||
"VFbELCWsd4dokrWDePdJI7TELpkgkfKJyACttKuYLUjE7kbwrsCPJ6wQ3EQo8KuoEPjgCOAsYFLK\n",
|
||||
"yOYHkY6klppMj6nNxhSl63AXIpafCh9qH4Vrk6fb6Xhn0o6wE0kiZlk01oO/l0qLZhqwHTw3SbuU\n",
|
||||
"aBWJm2TtBKniDiLwp5FdKrsV8clt9Jwl8JsRi2Y8cuUSj+CvAm7MaFvxENGTYUbn0o6jEbyidBm8\n",
|
||||
"7RDMosLL5YeEwrKDUsSwMFZ00qporEXTExG6fhAMBs8miU+hMmG6FKki8YxtE7NoasIaign8fOdE\n",
|
||||
"sQUYIJ1Fvb2ImD9jntts7o9DKg1tXXwPJNH7NqSnURrPIKNvR5hEr0bwitL98ZZVtkz2/s2pQW9E\n",
|
||||
"BA/FkqzraC8BbOdkKgYCefYkNdqMTO0B1HJgGUgEfzLZAr+FyAjmyMQyUGnRzETeh2vR/AYpF/+U\n",
|
||||
"8dJT8A4jlU8nm+RvD4rPhVAYjeAVpbnoagK/G6kHP4zs21LElrF1+acgycY4drDPHiSnUOu5f9cg\n",
|
||||
"J6Isgf85MumMi51Yxs4Z61o0x5n12YQrSK+b4wq2bHgQsWkepmLu33LQCF5RmotF5Pe/L5M8i2Yv\n",
|
||||
"EijayVSswJOSYLXY5ephz0BYjpgh8N594MXHfrg+fDyCn0BE4IPByEmk6Eh+K/A1sWdAI3hFaTK8\n",
|
||||
"z9R5g1bgU+YF9g5BsI+w9HAJErWDzBGxOaURlxX4khKsuRQQ+ERsC4ReiM9uI3N7PJ4lrKiZBiyt\n",
|
||||
"IhJ/CPg6MnitJgKvEbyiKFnsRKpgsgZI7SYUvkcIBT4pwWqxlTT1iuA3IvZLtQK/ALFiRgNbnD4+\n",
|
||||
"9opmLaFFMx05wRVlhfl7IzWooAEVeEVRsllBtF47CXdS+ceAGWZWqFNI77S4AmldXEINfBG8Q0gf\n",
|
||||
"nGoEGGSA15lE7RmQ8QkHkBPGViQPMbO69XsBMqjqXWgEryhK/fEC6eSYiRPBeweQhm2nIFUraRH8\n",
|
||||
"SkTg62XRICNkvWqH/y9CLJRYBY4XIJH7s6as8jnkPS+tXEUmtyLHQiN4RVG6JHuIDv+/H4l6TyRd\n",
|
||||
"4NcinvZg6mPRdBDvMBLFv5rKeS5WI2WhIGJ/OlVfIXi293sV01UWR5OsiqJ0FteDB/gn8L/IhNkp\n",
|
||||
"zc+8gxCsQxKx9Ry41RHuAz4B3B17/EXg2ZPTZuS9VGsBUcvGfhrBK4rSWT4K/MW5/0+k93la9G5Z\n",
|
||||
"iUyC0YUjeEAEvgcVEbzn7vdmxJNPm82uIajAK4rSSbx7pM1tO6uQiD6vudlK4Ei6fgT/IHCIbPHe\n",
|
||||
"BCxLmaSkYajAK4pSMl6AzFD1x5wFm0Tgvd1IP6CsAWab6ZA9U1vUg1cUpQZ4Hyuw0EqkfryrWzQU\n",
|
||||
"mOlrGdVPplJzVOAVRWkUK83/Lh7BF8G7ttF7kIRaNIqiNAor8E0QwTcnKvCKojQKO/1dN4jguyYq\n",
|
||||
"8IqiNAhvH1JtowJfI4oK/PnAQmAx0ZngLXORmWYeNX8fKWPnFEXp9qxELZqG0hMp/5kM9EaaCR0Z\n",
|
||||
"W2Yu0lMhi1o39H++MbfRO9CNmNvoHehmzC2+aPBxCM6o2Z50DzqsnUUi+DmIwK9AptX6KXBJwnJe\n",
|
||||
"R3dC6RBzG70D3Yi5jd6Bbsbc4ot6/wPeP2q2J89zigj8eMJm+SBTX42PLRMgzYUeB24Hjipl7xRF\n",
|
||||
"UZQOU6QOvsjlwSPARMRLuwC4BemNrCiKojSIIrbK6Ug7y/PN/Q8hE+x+NuM1y5H+yVudx5bQPlej\n",
|
||||
"oiiKUhA7+1VN6GU2MBnoQ3KSdTThyWIO4tcriqIoTcAFyMwmS5AIHuAq8wfwHuBJRPzvQ6J+RVEU\n",
|
||||
"RVEURVGalbyBUko2K4AnkEFkD5jHhgF3Ac8AdyLzRirJXAdsIDotWtbx+xDyXV0IvLRO+9gsJB1L\n",
|
||||
"H6muswMdL3Ce02OZzURkspSnEBfkavN403w/iwyUUrJZjnzgLp8DPmBu/xfwf3Xdo+bihcj8oK4o\n",
|
||||
"pR2/o5DvaG/kO7sEbenhknQsPwb8e8KyeizzGQOcYG63IFb4kTTR9/MM4A7n/gfNn1Kc5cDw2GML\n",
|
||||
"keQ2yJdkYV33qPmYTFSU0o7fh4heZd6B5pTiTKZS4P8jYTk9ltVzC/AvlPT9rIfyFxkopWQTAH9C\n",
|
||||
"pkCzEw+MRi6VMf9HJ7xOSSft+I1DvqMW/b4W49+QgY4/ILQT9FhWx2Tk6uh+Svp+1kPgtQdN53kB\n",
|
||||
"8sFfgFQsvTD2fIAe586Qd/z02GbzbWAKYjWsA76Ysawey2RagF8B1wDPxZ7r8PezHgK/FkkkWCYS\n",
|
||||
"PQMp+awz/zcBv0HGGmxALt0AxgIbG7BfzUza8Yt/XyeQPdmyIsfOitD3ke8n6LEsSm9E3G9ELBoo\n",
|
||||
"6ftZD4F/CJhBOFDqteR3nlRCBgCt5vZAJGs+HzmGbzKPv4nwi6EUI+343QpcjnxXpyDf3QcqXq24\n",
|
||||
"jHVuv4LQn9djmY+H2FoLgK84jzfV9zNpoJRSjClI1vwxpIzKHr9hiC+vZZL53Aw8i0yKvBp4C9nH\n",
|
||||
"77+R7+pC4Ly67mnXJ34s3wrcgJTxPo4IkZsP0mOZzVlI65fHCMtMz0e/n4qiKIqiKIqiKIqiKIqi\n",
|
||||
"KIqiKIqiKIqiKIqiKIqiKIqiKIqiKIqidBf+P41gkbjYnj6JAAAAAElFTkSuQmCC\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"<matplotlib.figure.Figure at 0x7f39dad22150>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(np.vstack([train_loss, scratch_train_loss]).clip(0, 4).T)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's take a look at the testing accuracy after running 200 iterations. Note that we are running a classification task of 5 classes, thus a chance accuracy is 20%. As we will reasonably expect, the finetuning result will be much better than the one from training from scratch. Let's see."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Accuracy for fine-tuning: 0.570000001788\n",
|
||||
"Accuracy for training from scratch: 0.224000000954\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"test_iters = 10\n",
|
||||
"accuracy = 0\n",
|
||||
"scratch_accuracy = 0\n",
|
||||
"for it in arange(test_iters):\n",
|
||||
" solver.test_nets[0].forward()\n",
|
||||
" accuracy += solver.test_nets[0].blobs['accuracy'].data\n",
|
||||
" scratch_solver.test_nets[0].forward()\n",
|
||||
" scratch_accuracy += scratch_solver.test_nets[0].blobs['accuracy'].data\n",
|
||||
"accuracy /= test_iters\n",
|
||||
"scratch_accuracy /= test_iters\n",
|
||||
"print 'Accuracy for fine-tuning:', accuracy\n",
|
||||
"print 'Accuracy for training from scratch:', scratch_accuracy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Huzzah! So we did finetuning and it is awesome. Let's take a look at what kind of results we are able to get with a longer, more complete run of the style recognition dataset. Note: the below URL might be occassionally down because it is run on a research machine.\n",
|
||||
"\n",
|
||||
"http://demo.vislab.berkeleyvision.org/"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 2",
|
||||
"language": "python",
|
||||
"name": "python2"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 2
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython2",
|
||||
"version": "2.7.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
|
|
@ -9,6 +9,7 @@ import hashlib
|
|||
import argparse
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from skimage import io
|
||||
import multiprocessing
|
||||
|
||||
# Flickr returns a special image if the request is unavailable.
|
||||
|
|
@ -27,6 +28,7 @@ def download_image(args_tuple):
|
|||
urllib.urlretrieve(url, filename)
|
||||
with open(filename) as f:
|
||||
assert hashlib.sha1(f.read()).hexdigest() != MISSING_IMAGE_SHA1
|
||||
test_read_image = io.imread(filename)
|
||||
return True
|
||||
except KeyboardInterrupt:
|
||||
raise Exception() # multiprocessing doesn't catch keyboard exceptions
|
||||
|
|
@ -48,6 +50,10 @@ if __name__ == '__main__':
|
|||
'-w', '--workers', type=int, default=-1,
|
||||
help="num workers used to download images. -x uses (all - x) cores [-1 default]."
|
||||
)
|
||||
parser.add_argument(
|
||||
'-l', '--labels', type=int, default=0,
|
||||
help="if set to a positive value, only sample images from the first number of labels."
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
np.random.seed(args.seed)
|
||||
|
|
@ -56,6 +62,8 @@ if __name__ == '__main__':
|
|||
csv_filename = os.path.join(example_dirname, 'flickr_style.csv.gz')
|
||||
df = pd.read_csv(csv_filename, index_col=0, compression='gzip')
|
||||
df = df.iloc[np.random.permutation(df.shape[0])]
|
||||
if args.labels > 0:
|
||||
df = df.loc[df['label'] < args.labels]
|
||||
if args.images > 0 and args.images < df.shape[0]:
|
||||
df = df.iloc[:args.images]
|
||||
|
||||
|
|
|
|||
|
|
@ -374,6 +374,7 @@ layer {
|
|||
type: "SoftmaxWithLoss"
|
||||
bottom: "fc8_flickr"
|
||||
bottom: "label"
|
||||
top: "loss"
|
||||
}
|
||||
layer {
|
||||
name: "accuracy"
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче