machinelearning_notebook/7_deep_learning/1_CNN/05-googlenet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GoogLeNet\n",
    "\n",
    "前面讲的 VGG 是 2014 年 ImageNet 比赛的亚军，那么冠军是谁呢？就是接下来要讲的 GoogLeNet，这是 Google 的研究人员提出的网络结构，在当时取得了非常大的影响，因为网络的结构变得前所未有，它颠覆了大家对卷积网络的串联的印象和固定做法，采用了一种非常有效的 Inception 模块，得到了比 VGG 更深的网络结构，但是却比 VGG 的参数更少，因为其去掉了后面的全连接层，所以参数大大减少，同时有了很高的计算效率。\n",
    "\n",
    "![inception2_improved.jpeg](images/inception2_improved.jpeg)\n",
    "\n",
    "这是 googlenet 的网络示意图，下面我们介绍一下其作为创新的 Inception 模块。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inception 模块\n",
    "\n",
    "在上面的网络中，我们看到了多个四个并行卷积的层，这些四个卷积并行的层就是 Inception 模块\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个 inception 模块的四个并行线路如下：\n",
    "1. 一个 1 x 1 的卷积，一个小的感受野进行卷积提取特征\n",
    "2. 一个 1 x 1 的卷积加上一个 3 x 3 的卷积，1 x 1 的卷积降低输入的特征通道，减少参数计算量，然后接一个 3 x 3 的卷积做一个较大感受野的卷积\n",
    "3. 一个 1 x 1 的卷积加上一个 5 x 5 的卷积，作用和第二个一样\n",
    "4. 一个 3 x 3 的最大池化加上 1 x 1 的卷积，最大池化改变输入的特征排列，1 x 1 的卷积进行特征提取\n",
    "\n",
    "最后将四个并行线路得到的特征在通道这个维度上拼接在一起，下面是PyTorch的实现一下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T12:51:05.427292Z",
     "start_time": "2017-12-22T12:51:04.924747Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.autograd import Variable\n",
    "from torchvision.datasets import CIFAR10\n",
    "from torchvision import transforms as tfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T12:51:08.890890Z",
     "start_time": "2017-12-22T12:51:08.876313Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义一个卷积加一个 relu 激活函数和一个 batchnorm 作为一个基本的层结构\n",
    "def Conv_ReLU(in_channel, out_channel, kernel, stride=1, padding=0):\n",
    "    layer = nn.Sequential(\n",
    "        nn.Conv2d(in_channel, out_channel, kernel, stride, padding),\n",
    "        nn.BatchNorm2d(out_channel, eps=1e-3),\n",
    "        nn.ReLU(True)\n",
    "    )\n",
    "    return layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T12:51:09.671474Z",
     "start_time": "2017-12-22T12:51:09.587337Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Inception(nn.Module):\n",
    "    def __init__(self, in_channel, out1_1, out2_1, out2_3, out3_1, out3_5, out4_1):\n",
    "        super(Inception, self).__init__()\n",
    "        # 第一条线路\n",
    "        self.branch1x1 = Conv_ReLU(in_channel, out1_1, 1)\n",
    "        \n",
    "        # 第二条线路\n",
    "        self.branch3x3 = nn.Sequential( \n",
    "            Conv_ReLU(in_channel, out2_1, 1),\n",
    "            Conv_ReLU(out2_1, out2_3, 3, padding=1)\n",
    "        )\n",
    "        \n",
    "        # 第三条线路\n",
    "        self.branch5x5 = nn.Sequential(\n",
    "            Conv_ReLU(in_channel, out3_1, 1),\n",
    "            Conv_ReLU(out3_1, out3_5, 5, padding=2)\n",
    "        )\n",
    "        \n",
    "        # 第四条线路\n",
    "        self.branch_pool = nn.Sequential(\n",
    "            nn.MaxPool2d(3, stride=1, padding=1),\n",
    "            Conv_ReLU(in_channel, out4_1, 1)\n",
    "        )\n",
    "        \n",
    "    def forward(self, x):\n",
    "        f1 = self.branch1x1(x)\n",
    "        f2 = self.branch3x3(x)\n",
    "        f3 = self.branch5x5(x)\n",
    "        f4 = self.branch_pool(x)\n",
    "        output = torch.cat((f1, f2, f3, f4), dim=1)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T12:51:10.948630Z",
     "start_time": "2017-12-22T12:51:10.757903Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input shape: 3 x 96 x 96\n",
      "output shape: 256 x 96 x 96\n"
     ]
    }
   ],
   "source": [
    "test_net = Inception(3, 64, 48, 64, 64, 96, 32)\n",
    "test_x = Variable(torch.zeros(1, 3, 96, 96))\n",
    "print('input shape: {} x {} x {}'.format(test_x.shape[1], test_x.shape[2], test_x.shape[3]))\n",
    "test_y = test_net(test_x)\n",
    "print('output shape: {} x {} x {}'.format(test_y.shape[1], test_y.shape[2], test_y.shape[3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到输入经过了 Inception 模块之后，大小没有变化，通道的维度变多了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们定义 GoogLeNet，GoogLeNet 可以看作是很多个 Inception 模块的串联，注意，原论文中使用了多个输出来解决梯度消失的问题，这里只定义一个简单版本的 GoogLeNet，简化为一个输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T12:51:13.149380Z",
     "start_time": "2017-12-22T12:51:12.934110Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class GoogLeNet(nn.Module):\n",
    "    def __init__(self, in_channel, num_classes, verbose=False):\n",
    "        super(GoogLeNet, self).__init__()\n",
    "        self.verbose = verbose\n",
    "        \n",
    "        self.block1 = nn.Sequential(\n",
    "            Conv_ReLU(in_channel, out_channel=64, kernel=7, stride=2, padding=3),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2)\n",
    "        )\n",
    "        \n",
    "        self.block2 = nn.Sequential(\n",
    "            Conv_ReLU(64, 64, kernel=1),\n",
    "            Conv_ReLU(64, 192, kernel=3, padding=1),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2)\n",
    "        )\n",
    "        \n",
    "        self.block3 = nn.Sequential(\n",
    "            Inception(192, 64, 96, 128, 16, 32, 32),\n",
    "            Inception(256, 128, 128, 192, 32, 96, 64),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2)\n",
    "        )\n",
    "        \n",
    "        self.block4 = nn.Sequential(\n",
    "            Inception(480, 192, 96, 208, 16, 48, 64),\n",
    "            Inception(512, 160, 112, 224, 24, 64, 64),\n",
    "            Inception(512, 128, 128, 256, 24, 64, 64),\n",
    "            Inception(512, 112, 144, 288, 32, 64, 64),\n",
    "            Inception(528, 256, 160, 320, 32, 128, 128),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2)\n",
    "        )\n",
    "        \n",
    "        self.block5 = nn.Sequential(\n",
    "            Inception(832, 256, 160, 320, 32, 128, 128),\n",
    "            Inception(832, 384, 182, 384, 48, 128, 128),\n",
    "            nn.AvgPool2d(kernel_size=2)\n",
    "        )\n",
    "        \n",
    "        self.classifier = nn.Linear(1024, num_classes)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.block1(x)\n",
    "        if self.verbose:\n",
    "            print('block 1 output: {}'.format(x.shape))\n",
    "            \n",
    "        x = self.block2(x)\n",
    "        if self.verbose:\n",
    "            print('block 2 output: {}'.format(x.shape))\n",
    "            \n",
    "        x = self.block3(x)\n",
    "        if self.verbose:\n",
    "            print('block 3 output: {}'.format(x.shape))\n",
    "            \n",
    "        x = self.block4(x)\n",
    "        if self.verbose:\n",
    "            print('block 4 output: {}'.format(x.shape))\n",
    "            \n",
    "        x = self.block5(x)\n",
    "        if self.verbose:\n",
    "            print('block 5 output: {}'.format(x.shape))\n",
    "            \n",
    "        x = x.view(x.shape[0], -1)\n",
    "        x = self.classifier(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T12:51:13.614936Z",
     "start_time": "2017-12-22T12:51:13.428383Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "block 1 output: torch.Size([1, 64, 23, 23])\n",
      "block 2 output: torch.Size([1, 192, 11, 11])\n",
      "block 3 output: torch.Size([1, 480, 5, 5])\n",
      "block 4 output: torch.Size([1, 832, 2, 2])\n",
      "block 5 output: torch.Size([1, 1024, 1, 1])\n",
      "output: torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "test_net = GoogLeNet(3, 10, True)\n",
    "test_x = Variable(torch.zeros(1, 3, 96, 96))\n",
    "test_y = test_net(test_x)\n",
    "print('output: {}'.format(test_y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到输入的尺寸不断减小，通道的维度不断增加"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T12:51:16.387778Z",
     "start_time": "2017-12-22T12:51:15.121350Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from utils import train\n",
    "\n",
    "def data_tf(x):\n",
    "    im_aug = tfs.Compose([\n",
    "        tfs.Resize(96),\n",
    "        tfs.ToTensor(),\n",
    "        tfs.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])\n",
    "    ])\n",
    "    x = im_aug(x)\n",
    "    return x\n",
    "     \n",
    "train_set  = CIFAR10('../../data', train=True, transform=data_tf)\n",
    "train_data = torch.utils.data.DataLoader(train_set, batch_size=64, shuffle=True)\n",
    "test_set   = CIFAR10('../../data', train=False, transform=data_tf)\n",
    "test_data  = torch.utils.data.DataLoader(test_set, batch_size=128, shuffle=False)\n",
    "\n",
    "net = GoogLeNet(3, 10)\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-22T13:17:25.310685Z",
     "start_time": "2017-12-22T12:51:16.389607Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0] Train:(L=1.329815, Acc=0.523318), Valid:(L=1.289094, Acc=0.566555), Time 00:01:15\n",
      "[ 1] Train:(L=0.868416, Acc=0.699808), Valid:(L=0.834760, Acc=0.715190), Time 00:01:15\n",
      "[ 2] Train:(L=0.661615, Acc=0.772998), Valid:(L=0.681946, Acc=0.765131), Time 00:01:15\n",
      "[ 3] Train:(L=0.538752, Acc=0.817315), Valid:(L=0.604022, Acc=0.794699), Time 00:01:15\n",
      "[ 4] Train:(L=0.443314, Acc=0.850264), Valid:(L=0.628162, Acc=0.788370), Time 00:01:15\n",
      "[ 5] Train:(L=0.377100, Acc=0.872462), Valid:(L=0.527649, Acc=0.825752), Time 00:01:14\n",
      "[ 6] Train:(L=0.310084, Acc=0.894981), Valid:(L=0.520545, Acc=0.833267), Time 00:01:15\n",
      "[ 7] Train:(L=0.263667, Acc=0.908628), Valid:(L=0.530805, Acc=0.839399), Time 00:01:14\n",
      "[ 8] Train:(L=0.214284, Acc=0.925831), Valid:(L=0.492261, Acc=0.850672), Time 00:01:14\n",
      "[ 9] Train:(L=0.178758, Acc=0.938679), Valid:(L=0.543371, Acc=0.843948), Time 00:01:14\n",
      "[10] Train:(L=0.154360, Acc=0.945213), Valid:(L=0.560078, Acc=0.839794), Time 00:01:14\n",
      "[11] Train:(L=0.127252, Acc=0.957121), Valid:(L=0.607742, Acc=0.833267), Time 00:01:14\n",
      "[12] Train:(L=0.122219, Acc=0.957980), Valid:(L=0.579313, Acc=0.842959), Time 00:01:14\n",
      "[13] Train:(L=0.100576, Acc=0.964734), Valid:(L=0.551588, Acc=0.856507), Time 00:01:14\n",
      "[14] Train:(L=0.085722, Acc=0.969969), Valid:(L=0.571536, Acc=0.851266), Time 00:01:14\n",
      "[15] Train:(L=0.078888, Acc=0.972746), Valid:(L=0.649491, Acc=0.847409), Time 00:01:14\n",
      "[16] Train:(L=0.079078, Acc=0.973026), Valid:(L=0.681464, Acc=0.840487), Time 00:01:14\n",
      "[17] Train:(L=0.069273, Acc=0.976582), Valid:(L=0.615183, Acc=0.848991), Time 00:01:15\n",
      "[18] Train:(L=0.062320, Acc=0.978780), Valid:(L=0.618147, Acc=0.858584), Time 00:01:16\n",
      "[19] Train:(L=0.060656, Acc=0.979220), Valid:(L=0.613905, Acc=0.857002), Time 00:01:17\n"
     ]
    }
   ],
   "source": [
    "res = train(net, train_data, test_data, 20, optimizer, criterion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyBUlEQVR4nO3deXxU9bn48c+TfV/IAgkJhE32hCUEBBEsVkEFrEXFulcvUrXV2v6q7b231fZ20VZt0SpCi6LiCirWulRQRARkBwFZQ0hCWJKQBLKRZb6/P84EQkhCgJycZOZ5v17zmplzzpx5chjOc77rEWMMSimlvJeP0wEopZRyliYCpZTycpoIlFLKy2kiUEopL6eJQCmlvJyf0wGcq9jYWJOSkuJ0GEop1aGsX7++wBgT19i6DpcIUlJSWLdundNhKKVUhyIi+5tap1VDSinl5TQRKKWUl9NEoJRSXq7DtREopdS5qq6uJjc3l8rKSqdDsV1QUBBJSUn4+/u3+DOaCJRSHi83N5fw8HBSUlIQEafDsY0xhsLCQnJzc+nRo0eLP6dVQ0opj1dZWUlMTIxHJwEAESEmJuacSz6aCJRSXsHTk0Cd8/k7vSYR7Dx0nN//ezvlVTVOh6KUUu2K1ySC3KJy5n65jy25JU6HopTyMsXFxTz33HPn/LmrrrqK4uLi1g+oAa9JBEO7RQOwIbvI4UiUUt6mqURQW1vb7Oc+/PBDoqKibIrqFK/pNdQpNIAesaFs2F/sdChKKS/zyCOPsHfvXoYMGYK/vz9hYWEkJCSwadMmtm/fzrXXXktOTg6VlZU88MADzJgxAzg1pU5paSmTJk3ikksuYeXKlXTt2pXFixcTHBzcKvF5TSIAGNotii925mOM8ZqGI6XU6R771za25x1r1X0OSIzgN5MHNrn+T3/6E1u3bmXTpk0sW7aMq6++mq1bt57s4jlv3jw6depERUUFI0aM4Pvf/z4xMTGn7WP37t28/vrrzJ07lxtuuIFFixZxyy23tEr8XlM1BDCsWzSFZVVkHy13OhSllBfLyMg4rZ//rFmzSEtLY9SoUeTk5LB79+4zPtOjRw+GDBkCwPDhw8nKymq1eLynRFBbzejgbAQXG7KL6B4T6nRESikHNHfl3lZCQ0+df5YtW8aSJUtYtWoVISEhjB8/vtFxAIGBgSdf+/r6UlFR0WrxeE+JYMtb9Hz3GgYHHNJ2AqVUmwoPD+f48eONrispKSE6OpqQkBB27NjB6tWr2zg6byoRJI8EYEqnHN7N7udwMEopbxITE8OYMWMYNGgQwcHBdO7c+eS6iRMnMnv2bFJTU+nbty+jRo1q8/i8JxHE9ILgTozy38sfD4yivKqGkADv+fOVUs567bXXGl0eGBjIRx991Oi6unaA2NhYtm7denL5z3/+81aNzXuqhkQgeSQ9KrdT6zJsztGBZUopBd6UCACSMwg9nkk0x3RgmVJKuXlZIrDaCSZF57JRE4FSSgHelggSh4KPHxNCs9iQXYwxxumIlFLKcd6VCAJCoEsqg1w7OFpWxf5CHVimlFLelQgAkkcSd2wrftSwfr9WDymllBcmggx8aioZHnhAG4yVUu1SWFgYAHl5eUybNq3RbcaPH8+6deta5fu8MBFYDcZXR2WzIbvY2ViUUqoZiYmJLFy40PbvsS0RiMg8ETkiIlubWH+ziGxxP1aKSJpdsZwmsitEJJHht4edh45RekLvWKaUstfDDz982v0IHn30UR577DEmTJjAsGHDGDx4MIsXLz7jc1lZWQwaNAiAiooKpk+fTmpqKjfeeGOrzjVk59Dal4BngZebWL8PGGeMKRKRScAcYKSN8ZySnEHKvlW4DGzJKWZ079g2+VqlVDvw0SNw6JvW3WeXwTDpT02unj59Og8++CD33nsvAG+99RYff/wxP/3pT4mIiKCgoIBRo0YxZcqUJqfIf/755wkJCWHLli1s2bKFYcOGtVr4tpUIjDHLgaPNrF9pjKmrpF8NJNkVyxmSRxJUfpAECrWdQCllu6FDh3LkyBHy8vLYvHkz0dHRJCQk8Ktf/YrU1FQuv/xyDhw4wOHDh5vcx/Lly0/efyA1NZXU1NRWi6+9TLZzF9D4ZBuAiMwAZgB069btwr8tOQOASVHZbMjuf+H7U0p1HM1cudtp2rRpLFy4kEOHDjF9+nQWLFhAfn4+69evx9/fn5SUlEann67PrhtqOd5YLCKXYSWCh5vaxhgzxxiTboxJj4uLu/Av7TIY/IK5LHQfG7OLdGCZUsp206dP54033mDhwoVMmzaNkpIS4uPj8ff35/PPP2f//v3Nfv7SSy9lwYIFAGzdupUtW7a0WmyOJgIRSQX+AUw1xhS22Rf7+kPX4Qyo2UFReTX7Csra7KuVUt5p4MCBHD9+nK5du5KQkMDNN9/MunXrSE9PZ8GCBfTr1/z0+D/60Y8oLS0lNTWVJ554goyMjFaLzbGqIRHpBrwD3GqM2dXmASRn0ClnFkGcYEN2MT3jwto8BKWUd/nmm1ON1LGxsaxatarR7UpLSwHr5vV1008HBwfzxhtv2BKXnd1HXwdWAX1FJFdE7hKRmSIy073Jr4EY4DkR2SQirTMyoqWSRyKuGkYF7dcGY6WUV7OtRGCMueks6+8G7rbr+88qaQQAkyKzeVGnmlBKeTHHG4sdExoDMX1I99nNrsPHdWCZUh7OWzqFnM/f6b2JACB5JMnlW3EZw+acYqejUUrZJCgoiMLCQo9PBsYYCgsLCQoKOqfPtZdxBM5IHkHApldJkUNs2F/EGB1hrJRHSkpKIjc3l/z8fKdDsV1QUBBJSec2PtfLE4E1o8VVkdnaYKyUB/P396dHjx5Oh9FueXfVUGxfCIxkXPA+NuboHcuUUt7JuxOBjw8kj6BvzXaKy6vJ1IFlSikv5N2JACB5JJHH9xJBmd6xTCnllTQRJGcgGEYHWfMOKaWUt9FE0HU4iA9XRmSzYX+x09EopVSb00QQGA6dBzLcZxe7jhznWGW10xEppVSb0kQAkDySxNJtiHHpwDKllNfRRACQPBK/mjL6+eRo9ZBSyutoIoBTdyyL0IFlSinvo4kAIKo7hHXmkqC9bMwuwuXSgWVKKe+hiQBABJIz6FO1nWOVNWQWlDodkVJKtRlNBHWSRxJWnkscxdpOoJTyKpoI6rgnoLskKFPbCZRSXkUTQZ2ENPAN4IoIvXWlUsq7aCKo4xcIiUNJYye7j5TqwDKllNfQRFBfcgZdSnfgb6rZlF3sdDRKKdUmNBHUlzwSH1cVA32ytHpIKeU1NBHUl1R/YFmxs7EopVQbsS0RiMg8ETkiIlubWC8iMktE9ojIFhEZZlcsLRbeGaJTuDhQB5YppbyHnSWCl4CJzayfBPRxP2YAz9sYS8slj6R35VaOV1azN18HlimlPJ9ticAYsxw42swmU4GXjWU1ECUiCXbF02LJGQSfKCBJCvSOZUopr+BkG0FXIKfe+1z3Mme5B5aNDdqrDcZKKa/gZCKQRpY1WikvIjNEZJ2IrMvPz7c3qvgBEBDG5WH7tcFYKeUVnEwEuUByvfdJQF5jGxpj5hhj0o0x6XFxcfZG5eMLSekMNjvYc6SUknIdWKaU8mxOJoL3gdvcvYdGASXGmIMOxnNKUgZxZbsJoZKNOVo9pJTybHZ2H30dWAX0FZFcEblLRGaKyEz3Jh8CmcAeYC5wr12xnLPkkYhxMdRnr1YPKaU8np9dOzbG3HSW9Qa4z67vvyBJ6QBcEbGfJdpgrJTycDqyuDHBURDXn1H+e9iUXawDy5RSHk0TQVOSM+hRuZ3SE1XsPqIDy5RSnksTQVOSRxJQfYxekqfjCZRSHk0TQVPcA8suDcpkg44wVkp5ME0ETYnpBcGd+E7oPi0RKKU8miaCpohA8kgG1u5gb34ZxeVVTkeklFK20ETQnOQMoiv2E8VxNuYUOx2NUkrZQhNBc9ztBMN997BR2wmUUh5KE0FzEoeCjx9XhOsEdEopz6WJoDkBIdAllRF+u9mUU0ytDixTSnkgTQRnkzySbhU7qDxRye4jx52ORimlWp0mgrNJzsCvtoJ+kq13LFNKeSRNBGdz2sCyYmdjUUopG2giOJvIrhCRxLiQfWzUgWVKKQ+kiaAlkjMYUPMtmQVlHC3TgWVKKc+iiaAlkkcSfuIQXSjkjbXZTkejlFKtShNBSyRnAHBr0mHmLs+k7ESNwwEppVTr0UTQEl0Gg18w13fOo6i8mpdX7Xc6IqWUajWaCFrC1x+6Die+eDPjLopj7pdaKlBKeQ5NBC2VnAEHN/Pg+GSOllXx6motFSilPIMmgpbqMRZcNQw9tIixfWKZszyT8iotFSilOj5NBC3V8zK4aBJ89jseHuFDoZYKlFIeQhNBS4nA5L+BfzCD1vySS3tHa6lAKeURbE0EIjJRRHaKyB4ReaSR9ZEi8i8R2Swi20TkTjvjuWDhneGqv0DuWv7QeRkFpVUsWK3jCpRSHZttiUBEfIG/A5OAAcBNIjKgwWb3AduNMWnAeOBJEQmwK6ZWMej70H8ySZue5obupbywfC8VVbVOR6WUUufNzhJBBrDHGJNpjKkC3gCmNtjGAOEiIkAYcBRo33UtInD10xAYzq9rnqWotIIFX2tbgVKq47IzEXQFcuq9z3Uvq+9ZoD+QB3wDPGCMcTXckYjMEJF1IrIuPz/frnhbLiwOrn6SsMIt/CF+KbO/yNRSgVKqw7IzEUgjyxre4utKYBOQCAwBnhWRiDM+ZMwcY0y6MSY9Li6uteM8PwO/BwOv4/rSBcSW7ea1NdpWoJTqmOxMBLlAcr33SVhX/vXdCbxjLHuAfUA/G2NqXVf9BZ/gKGaHzmXusp1UVmupQCnV8diZCNYCfUSkh7sBeDrwfoNtsoEJACLSGegLZNoYU+sKjYHJfyOlZi83VrzFa19rqUAp1fHYlgiMMTXA/cAnwLfAW8aYbSIyU0Rmujf7HTBaRL4BlgIPG2MK7IrJFv2uhtQb+bH/e3y+7FMtFSilOhwxpmG1ffuWnp5u1q1b53QYp6soompWBnvLAln73UXcNrav0xEppdRpRGS9MSa9sXU6srg1BEcT8L1n6e+TQ83nj2upQCnVoWgiaC0XXcnhXtO4rfZdliz5yOlolFKqxVqUCEQkVER83K8vEpEpIuJvb2gdT/y0v1Ds24mBXz9MZUWZ0+EopVSLtLREsBwIEpGuWI26dwIv2RVURyXB0Rwe/2d6kMueN3/ldDhKKdUiLU0EYowpB64DnjHGfA9r/iDVwICx3+M/wZPon/UyVVmrnA5HKaXOqsWJQEQuBm4G/u1e5mdPSB2biBA++U8cNJ2oeOseqCp3OiSllGpWSxPBg8AvgXfdYwF6Ap/bFlUHN6p/d+Z2+hmR5fupWfpbp8NRSqlmtSgRGGO+MMZMMcY87m40LjDG/MTm2DosEWHCpOuZX/NdfL+eDftXOh2SUko1qaW9hl4TkQgRCQW2AztF5P/ZG1rHNrZPLJ90mUke8Zj37oUq7UWklGqfWlo1NMAYcwy4FvgQ6AbcaldQnkBEuOeKNH56YgYUZcGSR50OSSmlGtXSRODvHjdwLbDYGFPNmVNKqwYu7RNLVdLFvOV7DayZA/uWOx2SUkqdoaWJ4AUgCwgFlotId+CYXUF5ChHhgcv78Juy6zgW0h0W3wfHDzsdllLexRg49A2ses56Vmc470nnRMTPPcNom2qXk841wxjDtc+tpHPJFl4wv0UCwmDaP6HHpU6HppTnqq2B7FWw49/Wo8Q9Rbz4wpgHYNzD4B/kbIxt7IInnRORSBF5qu52kSLyJFbpQJ2FiPDghD7851g3Phq1AIKj4eWp8MWfwXXGXTmVUuerqgy2vw/vzoS/9Ib518C6eRDfHybPgvvXQdpNsOIpmD0Gsr5yOuJ2o0UlAhFZBGwF5rsX3QqkGWOuszG2RnW0EgFYpYKpf/+Ko2VVfPbjdAI++hl88xb0mgDXzYHQWKdDVKpjKs2HXR/Bjg8h83OoqYSgKLhoIvS7yvo/Fhh2+mf2fgb/egCKsyH9h3D5YxB0xh1yPU5zJYKWJoJNxpghZ1vWFjpiIgBYtvMId7y4lptHduP/pg5ENr4MH/4CQmLg+heh2yinQ1SqYyjce6rKJ+drwEBksnWTqL5XQffR4HuWOTGryuDzP8Dq5yCsC1zzFPSd1CbhO6W5RNDSaSIqROQSY8wK9w7HABWtFaA3GN83npnjejH7i730iQ/jjjF3QOIwePt2ePEquPxRGP1jEHE6VKXan+pKWDkLti6C/B3Wsi6Drbr+fldbr8/l/05AKFz5exh4Hbz/Y3h9Ogz8Hkx6AsLi7fkb2rGWlgjSgJeBSPeiIuB2Y8wWG2NrVEctEQC4XIZ7Xl3P0m8PM++OEYzvGw+VJdYPcftiuGgSXPschHRyOlSl2o+j+6wLpoObIWXsqSv/6O6ts/+aKvjqb7D8CXeC+COkTfe4i7ILrhqqt6MIAGPMMRF50Bjz19YJseU6ciIAKDtRw7TZq8g9Ws47946mT+dwq3vbmjnwyX9DeAJc/xIkDXc6VKWc9+0H8N69IMC1z1tJwC75O62Lspyvodd34Jq/tl6yaQda7VaVxphj7hHGAA9dcGReKDTQj3/cnk6gvy93zV/H0bIq68pj5D3ww0+sjeZdCV+/YCWI1lJdYY1w7mD3qFZeqrbaujB682aI6Qn3LLc3CQDE9YU7P4ZJf4acNfDcxbD6eXB5/q1nL2QcQY4xJrmV4zmrjl4iqLMhu4jpc1YzJDmKV+8aSYCfOyeXH4X3fgS7PoYBU2HKMxAU2fzOGlNWCDmrrb7U2ashbxO4qmH0T+C7v/W4Yq/yIMU5sPBOyF0LGTPgiv8Dv8C2j+GDn8KeT6FrOkx91uqG2oG1WtVQg51mG2O6XVBk58FTEgHA4k0HeOCNTdyQnsTj309F6k7OLhesegaWPGYVTa+fDwmpTe/IGOtqP3vVqRN/wS5rnW8AdB1u9Uo6fhg2vwaj7oUr/6DJQLU/uz+Fd/7LGhA2ZRYMavMe6qcYA98shI8fhspjMPYhqztq7QmocT/qvz75vsrqxnradlXWs68/+IdYj4AQ8A8G/1DrOcD9fNr6Bq/P1huqGefda0hEjtP4nEICBLfgiycCfwN8gX8YY/7UyDbjgb8C/ljTW4872349xdQhXdl7pJRZn+2hT3w4/3VpT2uFj481+jEpw7oy+sflcNUTMOx26+RdWwOHt1on/LoTf+kh67NBkZA8yho40+1iSBx6agSlMdb61c9ZP8xJf7a+S3UMxnhu8q6tgWV/gC+fhM6DrIuf2N7OxiQCqddDr8vg41/CF49bjxZ91gf8gqwLMb8g8AuwXtdWW9W01eXuGYnP8UJ8zANWib6VnXeJ4Kw7FvEFdgHfBXKBtcBNxpjt9baJAlYCE40x2SISb4w50tx+PalEAFZPovtf38BHWw/xj9vSmdC/8+kblBVYV0h7P7MGydScsIrMVaXW+shu1tV+t1HWiT+uX/Mnd2Pg019bXfGG3W41iGkyaN8K9lglxM1vQHSKdVXa6zuQMsa6guzojh+ChXfB/hUw7DarC2d7/LvyNlpVt36B4BtoPdc9fBu89m1Bz3xjrP/P1eXuxFB+6vXJ9xVQXXbqdVK6lZjOgy1VQy340ouBR40xV7rf/xLAGPPHetvcCyQaY/6npfv1tEQAUFFVyw0vrCIzv5RF946mX5cGoxxdLutKacVT0KlXvRP/KIhMOvcvNAY++z/48i8w5GarHcLHt3X+GNV6ctfDV3+Fb/9lXU0Ous46ae5faVU7+AZag6d6uxND/ICOV2LI/AIW3WVdHV/9FAy5yemIPJZTiWAa1pX+3e73twIjjTH319vmr1hVQgOBcOBvxpiXG9nXDGAGQLdu3Ybv37/flpiddKikkql/X4Gfjw+L7x9DbFgjjWOtXTWw7HGrOD74erh2dsuuYpS9jIE9S6x+7VlfWlV5I+6GkTNPDXSqKofslbDnM9i79NQAq/AEKyH0+g70vAxCY5z7O87G5bIuRD7/A8ReBDfM7/CNse1da4wsPq/vbWRZw6zjBwwHJmC1OawSkdXGmF2nfciYOcAcsEoENsTquC6RQcy9LZ0bXljFPa+sZ8HdIwnyb3CV3tpXe+Mftk7+S38Lrhq4bu4FNUapC1BbDdvetRLA4a0QnghX/B6G3w6B4advGxACvS+3HgAluVbV4d7PrGkXNi0ABBKHnKpGSs5oP/+29as7B98A1zx95nxAqk3ZmQhygfrdS5OAvEa2KTDGlAFlIrIcSMNqW/A6qUlRPHXDEO5dsIFfvfMNT96QdqonkV3G/syqdvjP/1gno2kvWg1bqm1UlcGGV2DV362pkuP6wdTnrFJaS/8dIpOsuvVht1l93vM2wp6l1ol2xdPWlXdAuDX1ea/LrNJCTC9nqpH2r4KFP4TyQqt9avgdHa86ywPZWTXkh3VCnwAcwGos/oExZlu9bfoDzwJXAgHAGmC6MWZrU/v1xDaChp5ZupsnP93FLyb25d7xbdRz4usX4KNfWA3SN7zc9v22vU1ZgTWafM0cqCiyenpd8iD0ubJ1G+8riq074+1dalUl1c3LH5kMPce7q5HG2zutSU0V5H8LOz+CL56AqG5WVVBCmn3fqc7gSNWQMaZGRO4HPsHqPjrPGLNNRGa61882xnwrIh8DWwAXVhfTJpOAt7j/O73ZfaSUJz7eSc/YMCYO6mL/l468B3z84N8PwRs/gBtfbZ89Nzq6oixY+SxsfBVqKqw5c8Y8YN/ss8FRMGCK9TAGjmZaJYXMZdb8VhtfAcQ6KdeVFrqNOv8LgapyOLwNDm6y5gY6tAUOb7cGMwL0n2INzjqfQZLKNraVCOziDSUCgMrqWqbPWc3OQ8d5e+bFDOraRv9xNrxizbfS41K46Q2rPlqdH5fLugLP32U16OauterwxQdSb7Rmm43v51x8tTVWNVLm57D3c8hdY7UV+QVbvZHqEkPngY1X31QUW7d+rDvhH9xsDWQ07hsuBXeyEkxCmjUgMmEIdOqpVUEOcaTXkF28JREAHDleybXPfoUBFt83hviINrq13uY3rGkuuo2GH7ypDXlnU1sDRfusk33+TvdjBxTstq7664TGQ+oN1sjuyK7OxduUE8etu3bVJYaCndby0Hir+ihljFWlVXfSL8o69dnwxHonfPfJP6KrnvTbEU0EHdj2vGNMm72SPp3DeXPGqDN7Etnlm4XwzgxIGgE3v+0Vd3A6q+pKKNxjnSDz6z0K95yq+gCISLImMKt7xLqfO9r04iUHrCqkzM+t57J8a3l0Sr0r/TTokgZhcQ4GqlpCE0EH98m2Q8x8dT3XpCYya/oQ+3sS1dn2Liy62yrS37LIqm/2RscOWgO71r9kzSEDgFgnxLh+DU76F53Z3dMTuFxW+0JorPf+Djo4p8YRqFZy5cAu/OLKfjz+8Q6C/Hz443WD8fNtg2khBn4PfPzh7Tvg5alw67vOXdUaY82P1Ja9mY7lWd0v18+36s5Tb7RG8cb1hZje3tWY7uPj/Nw/yjaaCDqImeN6Ulldy9+W7qakoppZNw1tm2qi/tfA9AXw5i3w0tXQfzIER1s3CA+Odj+iTi071zEIxlh3aTt+yJo473i9R8P3NRVWu8WQH8DAa+278i45YCWADfOths+0m6zxFp162PN9SjlMq4Y6mHkr9vHbD7YzulcMc25LJyywjXL5niWw+H7rhNzcjIn+oQ0SRNSpJBEYbg0kaniiP1ndUk9AOIR3tqZNCOsM4V2s0sD296FwtzUlb/8pVlJIGds6fe9LcuHLp6wulcZlzcM09iGrCkipDk7bCDzMovW5/GLRFgYlRvDSnRlEh7bhSGBXrXUFX1lsDYSqKLK6EdY9N7rc/ag94T7Bdzn1CHOf7E8uc5/4m+qpZAzkrrOmUdj6DpwosQZHpd1k3Wc2pte5/03FOdaEfhtesd4PvRkuecijblOolCYCD/Tp9sPc99oGunUK4ZW7MkiI7AD11bXVrTvfTXWFe26d16xBUhhrKu4hP4AB1569p1NxtjWr68YF1vtht1oJIKrNb7ynlO00EXioVXsL+a+X1xEZ7M+rd4+kR2yo0yE5p+QAbHnTSgqFu61BUQPqqo4uPb3qqCjLSgCbXrMGdw27DS756flN6a1UB6GJwIN9k1vC7S+uwUdg/g8zGJjo5UP3jYED662qo28WWVVHEUnWPPe9JljLN7/uTgC3uxNAOxzcpVQr00Tg4fYcKeW2f37N8coa/nnHCDJ6dLCBS3aproCdH56qOjIu62Yuw++wJniLSHQ6QqXajCYCL3CguIJb//k1B4oqeP6WYXynX+ezf8ibHMuzZuHsMQ4iEpyORqk211wi0JvVeoiuUcG8fc/F9OkcxoyX17N40wGnQ2pfIhKtXkWaBJQ6gyYCDxITFsjr/zWK4d2jefDNTbyyKsvpkJRSHYAmAg8THuTP/B9mMKFfPP+7eBvPLN1NR6v+U0q1LU0EHijI35fnbxnOdUO78uSnu/jdB9/icmkyUEo1Tuca8lD+vj785fo0IoL9mffVPkoqqnn8+200WZ1SqkPRRODBfHyE30weQHRIAE8v2cWxymqevnFI281PpJTqEPTy0MOJCA9c3ofHpgxkybeHueKpL/h85xGnw1JKtSOaCLzE7aNTWDhzNCGBftz54loefGMjhaUnnA5LKdUOaCLwIsO7R/Pvn1zCAxP68O9vDvLdp5fz3sYD2qtIKS+nicDLBPr58tPvXsQHPx5Lt04hPPjmJu58aS0HiivO/mGllEeyNRGIyEQR2Skie0TkkWa2GyEitSIyzc541Cl9u4Sz6Eej+c3kAazZd5QrnvqC+SuztJupUl7ItkQgIr7A34FJwADgJhEZ0MR2jwOf2BWLapyvj3DnmB588uClDE/pxG/e38a02SvZffi406EppdqQnSWCDGCPMSbTGFMFvAFMbWS7HwOLAO3K4pDkTiHMv3MET92QRmZBGVfPWsHfluymqsbldGhKqTZgZyLoCuTUe5/rXnaSiHQFvgfMbm5HIjJDRNaJyLr8/PxWD1RZ3UyvG5bEkofGMXFQF55esovJz6xgY3aR06EppWxmZyKQRpY1rID+K/CwMaa2uR0ZY+YYY9KNMelxcXGtFZ9qRGxYILNuGso/b0/nWGU11z2/ksf+tY2yEzVOh6aUsomdQ0xzgfo3f00C8hpskw68ISIAscBVIlJjjHnPxrhUC0zo35mMHp144uOdvPhVFv/Zdpg/XDeYcRdpIlbK09hZIlgL9BGRHiISAEwH3q+/gTGmhzEmxRiTAiwE7tUk0H6EB/nzu2sH8fbMiwn09+H2eWv42VubKS6vcjo0pVQrsi0RGGNqgPuxegN9C7xljNkmIjNFZKZd36ta34iUTnz4k7Hcf1lv3tt0gMufWs7HWw86HZZSqpXorSrVOdmWV8IvFm5hW94xJg3qwmNTBxIfHuR0WEqps9BbVapWMzAxkvfuG8MvJvZl6Y4jfPep5Sxan6vTVCjVgWkiUOfM39eHe8f35sOfjKVPfBg/e3szd7yo01Qo1VFpIlDnrXd8GG/dczGPTh7A2ixrmopXVuk0FUp1NJoI1AXx8RHucE9TMax7NP+7eBvT565mX0GZ06EppVpIE4FqFcmdQnj5hxk8MS2VHQePMfGvy3nhi73U1Oo0FUq1d5oIVKsREW5IT2bJQ+MYd1Ecf/xoB9c9v5JvDx5zOjSlVDM0EahWFx8RxAu3DufZHwzlQFEFk59ZwVOf7uJETbMziSilHKKJQNlCRLgmNZFPHxrH5LREZi3dzeRnVvDx1oPUamOyUu2KJgJlq06hATx94xDm3ZFOZbWLma9u4DtPLuPlVVmUV+lEdkq1BzqyWLWZWpfhk22HmPtlJhuzi4kK8eeWkd25bXR3HZ2slM2aG1msiUA5Yv3+o8xZnsl/th/G38eHa4cmcvfYnlzUOdzp0JTySJoIVLu1r6CMeSv28fb6HCqrXYzvG8eMsT25uFcM7unJlVKtQBOBaveOllWxYPV+5q/KoqC0igEJEcy4tCdXpybg76tNWUpdKE0EqsOorK5l8aYDzP1yH3uOlJIQGcSdY1KYntGNiCB/p8NTqsPSRKA6HJfLsGzXEeYu38eqzELCAv2YPiKZu8b2ICEy2OnwlOpwNBGoDm3rgRLmfpnJB1sO4iMwbXgSM8f1ontMqNOhKdVhaCJQHiHnaDkvLN/LW+tyqal1MSUtkXsv6609jZRqAU0EyqMcOVbJ3C8zWfB1NuVVtVw5sDP3X9aHwUmRToemVLuliUB5pKKyKl5cmcVLX+3jWGUNl14Ux33jezGyZ4zToSnV7mgiUB7teGU1r67O5p8rMikorWJESjT3XdabcRfF6VgEpdw0ESivUFFVy5trs3lheSYHSyoZ3DWS+y7rxRUDuuDjowlBeTdNBMqrVNW4eHdjLs8v20tWYTl94sO497JeTE5NxE8Hpykv1VwisPV/hYhMFJGdIrJHRB5pZP3NIrLF/VgpIml2xqO8Q4CfDzeO6MaSh8bxt+lD8BHhp29u5rInlzFr6W69jaZSDdhWIhARX2AX8F0gF1gL3GSM2V5vm9HAt8aYIhGZBDxqjBnZ3H61RKDOlctlWLrjCHO/zGTNvqMADO4ayZS0RK5OTSAxSgeoKc/nSNWQiFyMdWK/0v3+lwDGmD82sX00sNUY07W5/WoiUBcir7iCf285yL+25LEltwSAESnRTElLZNLgBGLDAh2OUCl7OJUIpgETjTF3u9/fCow0xtzfxPY/B/rVbd9g3QxgBkC3bt2G79+/35aYlXfZV1DGB5vzeH9zHruPlOLrI4zuFcPktESuHNiFyGCd20h5DqcSwfXAlQ0SQYYx5seNbHsZ8BxwiTGmsLn9aolAtTZjDDsPH+df7qSQc7SCAF8fxvWNY0paIhP6xxMS4Od0mEpdkOYSgZ2/7lwgud77JCCv4UYikgr8A5h0tiSglB1EhH5dIujXJYKfX9GXzbklvL8pjw+25PHp9sME+/ty+YDOTE5NYHTvWMICNSkoz2JnicAPq7F4AnAAq7H4B8aYbfW26QZ8BtxmjFnZkv1qiUC1lVqXYW3WUd7fnMdH3xykqLwaH4EBiRGMSOlERkon0lM6EReu7Qqq/XNsHIGIXAX8FfAF5hljfi8iMwGMMbNF5B/A94G6Sv+apgKto4lAOaG61sWafUf5OrOQNVlH2ZRTTGW1C4AesaGMSIlmREonRqR0ontMiI5oVu2ODihTqpVV1bjYmlfC2n1HWZt1lLVZRZRUVAMQHx7oTgrRpKd0on9CBL46slk5TBOBUjZzuQx78ktZU5cY9h0lr6QSgPBAP4Z1j2ZESjQX94olLSlSRzirNqeJQCkHHCiuYO2+o6xxJ4bdR0oBKzGM7NmJ0b1iuaRPLH3iw7QqSdnOqV5DSnm1rlHBdB3alWuHWmMkj5ZVsWpvISv2FLBybwFLvj0CQFx4IGN6xTC6dyxjesfSVUc6qzamJQKlHJJztJyVewtYsaeQlXsKKCyrAqzG5zG9YxjTK5aLe8UQFRLgcKTKE2jVkFLtnMtlDWr7ak8BK/cWsjqzkPKqWkRgUGIko3vHMLpXLAmRQYQE+BIW6EdIgB8BftrWoFpGE4FSHUx1rYvNOcVWNdKeQjZkF1HjOvP/aoCvDyGBvoQG+FnJIbAuSfgSGuhHaICf+9mXsCA/LuoczuCkSCKCdPoMb6NtBEp1MP6+PqS7B6w9eDmUnahhU04xReVVlJ2ooexErfVcVfdcQ9mJGsqraik9UcORYycodS8vP1FLVa3rtP33igslLTmKIclRpCVF0S8hnEA/X4f+WuU0TQRKdQChgX6M6R173p+vqnFRUlHNtwePsTmnmM25JSzfVcA7Gw4AVsmif0I4ae7EkJYcRc/YUL2zm5fQqiGlvJQxhoMllWzOKWZTbjGbc4r5JreEsqpawOrmmpoceTIxpCVF0TkiULu6dlBaNaSUOoOIkBgVTGJUMJMGJwDW/Ep780vdpYZiNueUMGd55sn2ifBAP1JiQ0mJDaVHTEi916FEh2rvpo5KSwRKqWZVVtey/eAxtuQUk1lQxr6CMrIKyzhQVEH99uvIYP/TE0TMqSQRGaKN007TEoFS6rwF+fsyrFs0w7pFn7a8qsZF9tFy9heeSg5ZBeWszSpi8eY86l9jRodYSSI2LJDIYP9GHxEN3mvX2LajiUApdV4C/HzoHR9G7/iwM9ZVVteSc7T8ZILYV2AljJyj5WyrqKakovpkW0RTgv19z0gUfj6CwcowdYmmLt+cSjyNrwer1BIfHkhcvUd8eBBx4YFEBPl5bfuHJgKlVKsL8velT+dw+nQOb3Kb6loXx9xJodj9XPe+pNz9XO9xoLgCl7suquH5uu4ELiffN3hGcBnDzkPHyS89QVXN6d1pAQL9fOolh9OTRFxYIPERgUSHBBAR7E94oJ9H9ajSRKCUcoS/rw8xYYHEhLXtjX2MMRyrrCH/eCVHjp8gv96j7n1WQTlr9h2lqLy60X34CGdUZdU9okIaLguwnkP8CfTzwd/HB19fwc9H8PWxnp0uiWgiUEp5FRE5eZLuHd90iQWsdpDCshMcOWYliKLyqjNKKsXu0suBooqTJZvaRkaBN8dHwM/X57Tk4Ovjg79v/ffCTRnduHtszwv58xuliUAppZoQ4OdDQmQwCZEtnxHWGENZVS3F9ZOGO1lU1bqoqTXUugw1LkOty0V1g/fWs3EvP/W+xmWItan0pIlAKaVakYgQFmjN/ZQUffbt2wPtn6WUUl5OE4FSSnk5TQRKKeXlNBEopZSXszURiMhEEdkpIntE5JFG1ouIzHKv3yIiw+yMRyml1JlsSwQi4gv8HZgEDABuEpEBDTabBPRxP2YAz9sVj1JKqcbZWSLIAPYYYzKNMVXAG8DUBttMBV42ltVAlIgk2BiTUkqpBuxMBF2BnHrvc93LznUbRGSGiKwTkXX5+fmtHqhSSnkzOweUNTZ5RsNx1y3ZBmPMHGAOgIjki8j+84wpFig4z8+2hfYeH7T/GDW+C6PxXZj2HF/3plbYmQhygeR675OAvPPY5jTGmLjzDUhE1jV1Y4b2oL3HB+0/Ro3vwmh8F6a9x9cUO6uG1gJ9RKSHiAQA04H3G2zzPnCbu/fQKKDEGHPQxpiUUko1YFuJwBhTIyL3A58AvsA8Y8w2EZnpXj8b+BC4CtgDlAN32hWPUkqpxtk66Zwx5kOsk339ZbPrvTbAfXbG0MCcNvyu89He44P2H6PGd2E0vgvT3uNrVIe7eb1SSqnWpVNMKKWUl9NEoJRSXs4jE0F7nuNIRJJF5HMR+VZEtonIA41sM15ESkRkk/vx67aKz/39WSLyjfu71zWy3snj17fecdkkIsdE5MEG27T58ROReSJyRES21lvWSUQ+FZHd7udGb1Nytt+rjfH9WUR2uP8N3xWRqCY+2+zvwcb4HhWRA/X+Ha9q4rNOHb8368WWJSKbmvis7cfvghljPOqB1UNpL9ATCAA2AwMabHMV8BHWgLZRwNdtGF8CMMz9OhzY1Uh844EPHDyGWUBsM+sdO36N/FsfAro7ffyAS4FhwNZ6y54AHnG/fgR4vIm/odnfq43xXQH4uV8/3lh8Lfk92Bjfo8DPW/AbcOT4NVj/JPBrp47fhT48sUTQruc4MsYcNMZscL8+DnxLI9NqtHPtZY6oCcBeY8z5jjRvNcaY5cDRBounAvPdr+cD1zby0Zb8Xm2JzxjzH2NMjfvtaqwBnY5o4vi1hGPHr46ICHAD8Hprf29b8cRE0GpzHNlNRFKAocDXjay+WEQ2i8hHIjKwbSPDAP8RkfUiMqOR9e3i+GENUmzqP5+Tx69OZ+MeIOl+jm9km/ZyLH+IVcprzNl+D3a63111Na+JqrX2cPzGAoeNMbubWO/k8WsRT0wErTbHkZ1EJAxYBDxojDnWYPUGrOqONOAZ4L22jA0YY4wZhjVN+H0icmmD9e3h+AUAU4C3G1nt9PE7F+3hWP43UAMsaGKTs/0e7PI80AsYAhzEqn5pyPHjB9xE86UBp45fi3liIrBljqPWJCL+WElggTHmnYbrjTHHjDGl7tcfAv4iEttW8Rlj8tzPR4B3sYrf9Tl6/NwmARuMMYcbrnD6+NVzuK7KzP18pJFtnP4t3g5cA9xs3BXaDbXg92ALY8xhY0ytMcYFzG3ie50+fn7AdcCbTW3j1PE7F56YCNr1HEfu+sR/At8aY55qYpsu7u0QkQysf6fCNoovVETC615jNShubbBZe5gjqsmrMCePXwPvA7e7X98OLG5km5b8Xm0hIhOBh4EpxpjyJrZpye/Brvjqtzt9r4nvdez4uV0O7DDG5Da20snjd06cbq2244HVq2UXVm+C/3YvmwnMdL8WrLun7QW+AdLbMLZLsIquW4BN7sdVDeK7H9iG1QNiNTC6DePr6f7eze4Y2tXxc39/CNaJPbLeMkePH1ZSOghUY12l3gXEAEuB3e7nTu5tE4EPm/u9tlF8e7Dq1+t+h7MbxtfU76GN4nvF/fvagnVyT2hPx8+9/KW63129bdv8+F3oQ6eYUEopL+eJVUNKKaXOgSYCpZTycpoIlFLKy2kiUEopL6eJQCmlvJwmAqXakFgzo37gdBxK1aeJQCmlvJwmAqUaISK3iMga9xzyL4iIr4iUisiTIrJBRJaKSJx72yEisrrevP7R7uW9RWSJe/K7DSLSy737MBFZKNa9ABbUjYJWyimaCJRqQET6AzdiTRY2BKgFbgZCseY3GgZ8AfzG/ZGXgYeNMalYI2Hrli8A/m6sye9GY41MBWvG2QeBAVgjT8fY/Ccp1Sw/pwNQqh2aAAwH1rov1oOxJoxzcWpysVeBd0QkEogyxnzhXj4feNs9v0xXY8y7AMaYSgD3/tYY99w07rtapQArbP+rlGqCJgKlziTAfGPML09bKPK/DbZrbn6W5qp7TtR7XYv+P1QO06ohpc60FJgmIvFw8t7D3bH+v0xzb/MDYIUxpgQoEpGx7uW3Al8Y6x4TuSJyrXsfgSIS0pZ/hFItpVciSjVgjNkuIv+DdVcpH6wZJ+8DyoCBIrIeKMFqRwBriunZ7hN9JnCne/mtwAsi8lv3Pq5vwz9DqRbT2UeVaiERKTXGhDkdh1KtTauGlFLKy2mJQCmlvJyWCJRSystpIlBKKS+niUAppbycJgKllPJymgiUUsrL/X9TM38CkKoylAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEJCAYAAACZjSCSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvDklEQVR4nO3deZxcVZnw8d/T+55Ob+kt6U5C9j00IZAAUUQTEAKKGkDF6IggDMI77wzM6KijjqOvziIjEkUZZNgVkIgRECSBBAJZSHc6G9k66eo9nfS+VVed9497O13pdCdN0requu7z/XzqU3erqqduV9/n3nPOPUeMMSillHKvqFAHoJRSKrQ0ESillMtpIlBKKZfTRKCUUi6niUAppVxOE4FSSrmcY4lARB4RkXoRKR9ivYjIAyJyQETKRGShU7EopZQampNXBI8Cy8+wfgUwxX7cBjzkYCxKKaWGEOPUGxtj3hSR4jNsshJ4zFh3tG0WkXQRyTPG1JzpfbOyskxx8ZneViml1EDbtm07ZozJHmydY4lgGAqAyoB5j73stEQgIrdhXTUwYcIEtm7dGpQAlVIqUojIkaHWhbKyWAZZNmh/F8aYXxljSowxJdnZgyY0pZRS5yiUicADjA+YLwSqQxSLUkq5VigTwVrgi3brocVA89nqB5RSSo08x+oIROQpYBmQJSIe4DtALIAxZg2wDrgaOAB0AKvP9bO8Xi8ej4eurq7zDTvsJSQkUFhYSGxsbKhDUUpFCCdbDd10lvUGuHMkPsvj8ZCamkpxcTEig1U9RAZjDI2NjXg8HiZOnBjqcJRSESIi7izu6uoiMzMzopMAgIiQmZnpiisfpVTwREQiACI+CfRxy/dUSgVPKO8jUEqpiOTzG9q6emnp8tLS5aW1q5fWrl68Pj8+v8FvDD6/OTnd6zf47XmfwZq2t+mb9vsNJcUZXD515JvQayIYAU1NTTz55JN8/etf/1Cvu/rqq3nyySdJT093JjCl1DkzxtDW3Ut9azf1Ld00tHXT3OmlpbPvwG499x/o7flOL+09PkdiumPZZE0E4aqpqYlf/OIXpyUCn89HdHT0kK9bt26d06EppQbw+Q2N7f0H94aWbupbu2ho7bYO+q3d9nQXXV7/oO8RGy2kJcSSmhBDqv2cnZVyynxaov0csCwuJopoEaKihGgRoqNOnY62p6OisNZJ4DLnioU1EYyA+++/n4MHDzJ//nxiY2NJSUkhLy+PHTt2sHv3bq6//noqKyvp6uriG9/4BrfddhsAxcXFbN26lba2NlasWMHSpUt5++23KSgo4MUXXyQxMTHE30yp4PD6/DR1eGnq6OFEh5cTHT2caLemrWXWdK/Pj9+A3xiM/Ww9rDP4wHWB8367uOV4Rw+Nbd34B+nDIC0hhpy0BLJT4lkwIZ3slHhy0uLJSU0gOzWe7NR40pNiSUuIJT4mKqLq6yIuEfzLH3exu7plRN9zZn4a37l21pDrf/SjH1FeXs6OHTtYv34911xzDeXl5SebeD7yyCNkZGTQ2dnJRRddxKc//WkyMzNPeY/9+/fz1FNP8fDDD/PZz36W5557js9//vMj+j2UCrbWLi9HGjs4fKydo8c7ONbWTVOHl+PtPacc9Fu7eod8j7iYKMYmxZKeGEd8rHUAjhKIsp9PzkdFIfbyobZZkJROdmo8OfaBPTs14eR0QuzQV++RLuISQThYtGjRKe38H3jgAV544QUAKisr2b9//2mJYOLEicyfPx+ACy+8kIqKimCFq9R5ae70cqSxnYrGDiqOtVPR2M4Re7qxveeUbVPiY0hPiiUjOY70pDiKs5IZmxRnPZJjSU+KY2xSrD1vTSfGRkfU2Xc4irhEcKYz92BJTk4+Ob1+/Xpee+013nnnHZKSkli2bNmg9wHEx8efnI6OjqazszMosSo1HJ09Pj6oa6WisZ3Dx+wDfWM7FcfaOdHhPWXbvDEJFGUm8fFZ4yjKTKY4M4mizGSKMpNIiou4Q05E0L/KCEhNTaW1tXXQdc3NzYwdO5akpCT27t3L5s2bgxydUsNnjKGmuYs9NS32o5U9tS1UHGs/Wa4uAvljEinKTGLFnLyTB/pi+2Dv5iKW0UoTwQjIzMxkyZIlzJ49m8TERMaNG3dy3fLly1mzZg1z585l2rRpLF68OISRKtWvy+tjf10be2pa2G0f+PfWttLc2X+GPz4jkRm5aVw7N58ZealMzk5hfIYe7CONWF3+jB4lJSVm4MA0e/bsYcaMGSGKKPjc9n3V6Yx9E5LX58fba+jx+en19097Ax49vdZ2nV4fBxvarLP8mhYONbSdPMtPjI1mWm4qM/LSmJlnPU/LTSU1QTs3jBQiss0YUzLYOr0iUCqM9fT6eedQIy+X17B+XwMtnV68Putgf64K0hOZkZfKitm5zMhLY0ZeGkUZSY62U1fhTROBUmGmy+tjwwcNvFJey2t76mjp6iU5LporpmWTNyaR2Ogo4qKF2OgoYmOiTp23l50yHx1FXIwQFx3NhMwkxiTqWb46lSYCpcJAa5eXN/Y18HJ5DW/sbaDT62NMYiwfn5XL8lm5LJ2SpeXyyjGaCJQKkRPtPfxlTx2vlNfy1v5j9Pj8ZKXE86mFBayYncfFkzKIjY6YDoJVGNNEoFQQ1bd08cquWl7eVcvmQ8fx+Q0F6Yl84ZIils/OZeGEsURrWb0KMk0ESjmktcvLB3Wt7KlpZV9tKzurmin1NGEMTMpK5muXT2LF7DxmF6TpnbMqpDQRhEBKSgptbW1UV1dz99138/vf//60bZYtW8ZPf/pTSkoGbe2lwojX5+fwsXb21rayr7aFfbXWwb+qqf/u8JT4GKbnpnLvx6ayfHYuU3JS9OCvwoYmghDKz88fNAmo8GSMoa6lm721LfZBv5W9ta0crG872ZwzOkqYnJ3MwqKx3HzxBKbnpjItN5WC9EQ98KuwpYlgBNx3330UFRWdHI/gu9/9LiLCm2++yYkTJ/B6vfzgBz9g5cqVp7yuoqKCT37yk5SXl9PZ2cnq1avZvXs3M2bM0L6GwoTfb3j38HHWllbz6q7aUzpRy01LYFpuKpdPyWJ6XirTxqUxOSeZ+Bht3aNGl8hLBH++H2p3jux75s6BFT8acvWqVau45557TiaCZ599lpdffpl7772XtLQ0jh07xuLFi7nuuuuGPCt86KGHSEpKoqysjLKyMhYuXDiy30ENmzGGMk8za0ureamsmrqWbhJjo/nYzHGUFI1lWm4q03NTSU+KC3WoSo2IyEsEIbBgwQLq6+uprq6moaGBsWPHkpeXx7333subb75JVFQUVVVV1NXVkZubO+h7vPnmm9x9990AzJ07l7lz5wbzKyhgf10ra0ur+WNpNRWNHcRGC1dMzeGb1+TzsRk52nOmiliR98s+w5m7k2688UZ+//vfU1tby6pVq3jiiSdoaGhg27ZtxMbGUlxcPGj304G0DDn4Ko938MeyatbuqGZvbStRApdMzuSOZZNZPiuPMUl6F66KfJGXCEJk1apVfPWrX+XYsWNs2LCBZ599lpycHGJjY3njjTc4cuTIGV9/+eWX88QTT/CRj3yE8vJyysrKghS5+zS0dvOnsmrWllaz/WgTAAsmpPOda2dyzdw8clITQhugUkGmiWCEzJo1i9bWVgoKCsjLy+OWW27h2muvpaSkhPnz5zN9+vQzvv6OO+5g9erVzJ07l/nz57No0aIgRe4OXV4ffyyt5sUd1bx98Bh+A9NzU/n7T0zjunn5jM9ICnWISoWMdkM9Crnt+56P5k4vj28+wv9sOsyxth4mZCRx3bx8rpufz9RxqaEOT6mg0W6olevUNnfxm42HePLdo7T3+Lh8aja3XzGJSyZlal2MUgNoIlAR5UB9G7968yAvvF+Fz2/45Nx8vnbFJGbljwl1aEqFrYhJBMYYV5zpjbaivGDZfvQEa9Yf5C976oiLjuKmRRP46mWTtOxfqWGIiESQkJBAY2MjmZmRfdlvjKGxsZGEBG3VAtb+WP9BA2vWH+Tdw8cZkxjLXR+5gFsvLSYrJT7U4Sk1akREIigsLMTj8dDQ0BDqUByXkJBAYWFhqMMIqV6fnz/trGHNhkPsqWkhb0wC37pmBjctmkByfET8pJUKqoj4r4mNjWXixImhDkM5rLPHx7NbK3n4rUN4TnQyJSeFn35mHtfNyycuRgdwUepcRUQiUJHLGMOu6hae2+7hD+9XcaLDy4VFY/nOtbO4cnqODriu1AjQRKDCUn1LF3/YUcVz26rYV9dKXHQUH5uZw+olE7moOCPU4SkVURxNBCKyHPgZEA382hjzowHrxwKPAJOBLuDLxphyJ2NS4avL6+OVXbU8v72Kt/Y34DdW1w8/uH42187N135/lHKIY4lARKKBB4GrAA+wRUTWGmN2B2z2T8AOY8wNIjLd3v5Kp2JS4ccYw5aKEzy/3cOfympo7e6lID2ROz9yATcsKGBSdkqoQ1Qq4jl5RbAIOGCMOQQgIk8DK4HARDAT+DcAY8xeESkWkXHGmDoH41Jh4GhjB8+/7+H57VUcPd5Bclw0K+bk8emFhVw8MUPL/pUKIicTQQFQGTDvAS4esE0p8Clgo4gsAoqAQuCURCAitwG3AUyYMMGpeJXDWrq8rCur4fntVbxXcRwRWDI5i3uvmsInZuVqf/9KhYiT/3mDndINvC32R8DPRGQHsBN4H+g97UXG/Ar4FVidzo1smMpp3b0+Hn7zEA++cZBOr4/J2cn8w/Jp3LCggLwxiaEOTynXczIReIDxAfOFQHXgBsaYFmA1gFi3BB+2HypCbNx/jG+/WM6hY+1cPSeXr10+mbmFYyL6DnClRhsnE8EWYIqITASqgFXAzYEbiEg60GGM6QH+BnjTTg5qlKtr6eL7L+3mpbIaijOT+O2XF3HF1OxQh6WUGoRjicAY0ysidwGvYDUffcQYs0tEbrfXrwFmAI+JiA+rEvkrTsWjgqPX5+exd47wH3/5gB6fn3s/NpWvXTGJhNjoUIemlBqCo7Vzxph1wLoBy9YETL8DTHEyBhU8246c4Ft/KGdPTQtXTM3meytnUZSZHOqwlFJnoc001Hk70d7Dj1/ey9NbKskbk8Cazy/kE7NytR5AqVFCE4E6Z36/4XfbKvnRn/fS2tXL1y6fxN1XTtEeQJUaZfQ/Vp2T3dUtfOsPO9l+tIlFxRl8//rZTMvVMYCVGo00EagPpbXLy3/+ZT+Pvn2YsUlx/Ptn5vGphQVaDKTUKKaJQA2LMYaXymr4/ku7aWjr5paLJ/D3H5+uHcEpFQE0EaizOlDfyrdf3MXbBxuZUzCGh79Ywrzx6aEOSyk1QjQRqCG1d/fywF/385u3DpMUF833V87i5ouLiNYO4ZSKKJoI1GmMMazbWcv3X9pNbUsXny0p5L7l08nUAeGVikiaCNQpDtS38d21u9h44Bgz89J48JaFXFg0NtRhKaUcpIlAAVYx0H//9QC/2XiIxNhovrdyFrdoMZBSrqCJwOWMMfy53CoGqmnu4jMXFnLfiulkaTGQUiPHGDh+CHq7IS4JYpOt55hEiIoKdXSaCNzsYINVDPTWfqsY6Oc3L+DCIh0YXg3C1wue9yAuGTImQbzePHhWTUfh0AY4/Kb1aKsdfLvYJOsRmCCGWjbxcrhg5Efz1UTgQh09VjHQr986REJsNP9y3SxuuXgCMdGhPzNRYcQYqC2D0mdg57PQ3tC/LjkbMiZbSSFjEmRO6p9OGBO6mEOprQEO9x34N8CJCmt5crZ1AC++DBLToacDvB3Q0z7guQO87eDttKbbak/fNipaE4E6P8YYXraLgaqbu7jxQqs1UHaqFgOpAC011oG/9Gmo3w1RsTBtOcy+EUSsIo7Gg3D8MBxaD6VPnvr6pMwBSWIyZEy0phMjqOFBVzNUbOo/8Nfbw7HHp0HxUrj4DisB5Myw9ttIMM4M0KiJwCXqW7v4u2dLeWv/MabnpvLATQsoKXZBMZAx1j/ovnVw5G1IzYfc2TBuFoybDUku2AfD0dMOe/8EpU9ZB3fjh8JFcM1/wKwbzryfetqts9+TCeKQ9ah4C8qePnXbuBRIzYXUPEjLt6fzIS3Pek7NtR7RYXjHurcTjm7uP/BXv2/tp5hEmLAY5nwGJl0BufMg2qFDq0NduYhxKMM4paSkxGzdujXUYYwqBxvauPWR92hs6+Eflk/jC4uLIrsYyOeFI5tg35+tBNB01FqeM9Mq3ggs4kgr6E8K42ZB7hzrbNapf+Rw4vfDkY3Wmf/uF6GnDdInwNxVMG+VdSZ/vryd1pVDX3JoqYbWamitta48WmvA7x3wIrGKUwKTQ1q+lTyyp1l/x/iU84/tbNrqofJd6+Bf+S5U77BijYqBghLrbH/SFVB4EcSE/1W1iGwzxpQMts4Fv3Z32370BF95dAtRIjx92+LI7RqiswkOvGYd/Pf/BbqbISYBJi2Dy/4Opi63DigArXVQV24/dkFtORz8K/h7rfUxCZA93b5ymN2fJCLl6qHhA+vMv+xZaPFYRRmzP2UlgAmXjGwrlthEGDfTegzG74eORishtNbYiaLWShYtNdDssSqpOxoDXiRWksqdYz/mWs8p4879jNnvh2P7+g/6RzfDCXv49Oh4yF8Al3zdKuefsDjiKsv1iiCC/WV3HX/71HbGpSXw2JcXRd5oYSeO9J/1H9lkHciTsqzy7GlXW0kgbpjfubcbjn1gJYW+JFFbDh3H+rdJzYexRdZVxJgCSCu0zlT7ppOzHLt0Py/GQOMBOPA6lD0D1dtB7ErHeausfRWbGOooz6y3G1qqoH4v1O60KrFrd0LTkf5tkrNPTw6ZF1gVrAP1dEDVNuugX/kuVL4HXU3WuqQsGH8xTLgYxi+G/Pmj4oz/bM50RaCJIEI9+e5RvvWHncwuGMMjX7ooMu4L8Puh5n3r4L93HdTvspZnTYNpK6wDWmHJ4P/456q1Dup2WlcOdbutM9QWj3Xm6us5ddvoOCsxpBXayaGg/zmtAMYUBu+qoqXabrq4wXpurbaW586BeTdZFb+p44ITi5M6m+yrup39CaJhb//fJsa+IsmdA9kzrMRxdLO1Xd8VYNa0/oP+hMVWpXY4JvTzpInARYwx/Odr+3ng9f0sm5bNgzcvHJ0jhnU1Q8M+65+6fq/1XLsT2utBomDCpfbBf8XIlGV/WH6/dbXQUgXNVfaznSD6lrVW9x9s+iRnW2Xc42bZzzOtA1Rc0vnF03kCKjZaB/1D66Fxv7U8MaO/LHviFaHZV8HW22Nf3e089eqhq8kq9stf2H/gH78ocor8zkITgUv0+vx884VyntlayWdLCvnXG+YQG+6Vwp1N9gF/j/Vcbz/3ncGC9c+bNdU6cE7+KEy5anT88/p9VsV0c5V1FdFUaX3Put3W9+zttDcU6yx03EzImWWXqc+GscVDX914O+HoO/1n/TWlVguW2CQoutQqFpt4hfU+YXDnasgZA211VmKMiQt1NCGhlcUu0NHTy51PbOeNfQ3c/dELuPeqqeE1alhPu1XmXr/71AN/a03/NjGJVquQiZfbrUNmWM/pRSNb3BMsUdH9zSG58NR1fp/V5LJul7VP6sqtBLHnJcA+OYtJhJzp/ckhY7J1Znt4g1Wu7euxWrAUXgSX/4N11l9Q4toD3RmJ9DcWUKfRK4IIcKytm688uoWdVc384Po53HzxhNAG5O20DvrV7/c/ju2zzljBOmvNnma1zOl75EyHMRP07LWnwyoGO5kgdlmPwErrcXP6i3qKLg1OU0o16ukVQQQ70tjOrY+8R21LF7/8QglXzQxyBWBvj1Vp23fAr3rfOoAZn7U+Odsqk5250mp9kTMTxozXA/5Q4pKgYKH1CNRWb7X8yZpqtU5SagRpIhjFyjxNrP6fLfiN4Ym/Wez8uAE+r3W2GnimX7erv4VG4lirvfXUe6zn/AVWa5lwKqIarVJyrIdSDtBEMEq9sa+eO5/YTkZyHL/98iImZztcPPDBq/DcV6C7xZqPT7PO8Bff0X/QTy/Sg75So5AmglHod1sruf/5nUwbl8qjqy8iJy3B2Q/c/xo8c4tVln/p3VaxxdiJWryjVITQRDCKGGN48I0D/PTVD1h6QRYPfX4hqQkOd8514HV4+marcveLL46OZptKqQ9FE8Eo4fMbvrO2nMc3H+WGBQX8+NNziYtx+Iz84BtWEsiaCl9cq0lAqQiliWCU+Ld1e3h881G+dsUk7vvEdKKcHkv40AZ46iar7bpeCSgV0TQRjALPbDnKrzce5kuXFvOPK2Y4/4GH34InP2fd2XrrWkjOdP4zlVIho7V9Ye7dQ4186w/lXDYli29dE4QkULEJnvys1S/9rWu1zbpSLqCJIIwdbezg9se3MT4jiZ/fvND5wWSOvANPfMbqJfPWP2q7daVcQhNBmGrt8vKV327Bb+CRWy9iTKLDrYOOvgtP3GiNCnXrHyOji2Kl1LA4mghEZLmI7BORAyJy/yDrx4jIH0WkVER2ichqJ+MZLXx+w98+9T6Hj7Xz0OcXUpzl8IAylVvg8U9bIzzd+pJ2zqWUyziWCEQkGngQWAHMBG4SkYHj1d0J7DbGzAOWAf8uIq7vOvGH6/awfl8D/7JyFpdOdriM3rMNHv+UVRfwpZesKwKllKs4eUWwCDhgjDlkjOkBngZWDtjGAKli9ZecAhwHBozk4S7PbDnKb+wWQrdcXOTsh1Vth/+9wWoa+qWXrNG1lFKu42QiKAAqA+Y99rJAPwdmANXATuAbxvT1VdxPRG4Tka0isrWhocGpeENus91C6PKp2c63EKreAf97PSSOsYqDxhQ6+3lKqbDlZCIY7I6ngYMffALYAeQD84Gfi0jaaS8y5lfGmBJjTEl2dvZIxxkWjjZ2cMfj25iQkcR/37Tg9BZCHcdh8xoof94autHnPfcPqymDx1ZaHcfd+hKkjz+/4JVSo5qTN5R5gMAjTCHWmX+g1cCPjDU6zgEROQxMB95zMK6w02K3EDLAbwZrIVRbbnX6dqKif1lULGRNsUbxyplh9fOfMwPSi8/cGVxtuZUE4lKs4qCxDhc/KaXCnpOJYAswRUQmAlXAKuDmAdscBa4E3hKRccA04JCDMYUdn99wt91C6LGvLDq9hVD58/DindbZ+5fWQXyqPaC7Pe6tZwuUP9e/fd9wj32Joe85Ld/a/rHrIDYRvvRH685hpZTrOZYIjDG9InIX8AoQDTxijNklIrfb69cA3wceFZGdWEVJ9xljjg35phGor4XQv94w+9QWQn4fvP4vsOlnMP5i+Oxj/c068+ae+ibdrfbA77utYqP63XDwr1D6ZP828WnWAN7xKdZ9AhmTnP9ySqlRwdG+howx64B1A5atCZiuBj7uZAzh7On3hmgh1HHcGgTm4F+h5Muw/MdnHpA8PhUKS6xHoI7j1lVA39VD53H4yLcgc7IzX0gpNSppp3MhMmQLobpdVtfPLdVw7QNw4a3n/iFJGVC8xHoopdQQNBGEwJHGdu54fBtFmUn8/OaAFkID6wPGXxTaQJVSrqCJIMisFkJbT7YQSkuItesDvgeb/uv0+gCllHKYJoIg6vX5+dsn36cisIVQx3F47m/g4OvDqw9QSqkRpokgiH64bi8bPmjghzfMsVoIjWR9gFJKnSNNBEHy+p46Htl0mNVLirn54glaH6CUChuaCILA7zf89NUPKM5M4p+WT4W/fEfrA5RSYUMTQRC8uruWPTUtPLCyiNinP6f1AUqpsKKJwGF+v+G/XtvPJRntXPveLVofoJQKO2dNBCKSDHT2dQ8tIlFAgjGmw+ngIsHLu2o5UHuC9/J+jnQchy/9CcYvCnVYSil10nC6oX4dSAqYTwJecyacyOL3G3722n5+kPYHMk6UwnUPaBJQSoWd4SSCBGNMW9+MPZ10hu2VbV15DbkNG1nV87xVJzDrhlCHpJRSpxlOImgXkYV9MyJyIdDpXEiRwec3PP7qZn4W/xBm3Cz4xA9DHZJSSg1qOJXF9wC/E5G+QWXygM85FlGEeKm0km80/4SUOC9y46PWGABKKRWGzpoIjDFbRGQ61qAxAuw1xpzHOImRz+c3nPjzD1kZvRv/J38B2VNDHZJSSg3prEVDInInkGyMKTfG7ARSROTrzoc2er39+ot8oftpPBNWErXgllCHo5RSZzScOoKvGmOa+maMMSeArzoW0SjX21LPjLfvpTo6n/ybHwx1OEopdVbDSQRRIiJ9MyISDejtsIPx+zn2+JdJ9bdy9KMPEpWQGuqIlFLqrIaTCF4BnhWRK0Xko8BTwJ+dDWt08r393+TWv8XDyV/lkkuXhTocpZQaluG0GroPuA24A6uy+H2slkMqUOUW5PXvsc63iKlX301UlJz9NUopFQbOekVgdy2xGTgElABXAnscjmt06WzCPPdl6sjgt1n/h6tmaW+iSqnRY8grAhGZCqwCbgIagWcAjDEfCU5oo4QxsPZvMc3V3NH1be66aiEBVSpKKRX2zlQ0tBd4C7jWGHMAQETuDUpUo8mWX8OetTwUcyv+ggu5ckZOqCNSSqkP5UxFQ58GaoE3RORhEbkSq45A9akpg1e+SXX2Un7adhX3fmyqXg0opUadIROBMeYFY8zngOnAeuBeYJyIPCQiHw9SfOGruw1+vxqTlMFXW/6GeeMzWDYtO9RRKaXUhzacyuJ2Y8wTxphPAoXADuB+pwMLe3/6Ozh+iNdm/Cu7muO452NT9GpAKTUqDec+gpOMMceNMb80xnzUqYBGhR1PQtnT9F7293ynNJ0FE9K5YqpeDSilRqcPlQgU0LDPuhoovoynE1dR3dyldQNKqVFNE8GH4e2E362G2CS6rlvDz984TEnRWC6bkhXqyJRS6pxpIvgwXv5HqN8FN/ySZ/b2UtvSxb1X6dWAUmp000QwXEfehm3/A0u+QVfxR/jF+gMsKs7g0smZoY5MKaXOiyaC4frgFYiKhSvu46n3jlLX0s09V2lLIaXU6KeJYLiObIKCC+mSBH6x/iAXT8zg0slaN6CUGv00EQxHdxtUbYfiJTy++QgNrd3ce5UOP6mUigyOJgIRWS4i+0TkgIicdhOaiPy9iOywH+Ui4hORDCdjOieVm8H46C68lDUbDnHp5EwWT9K6AaVUZHAsEdgjmT0IrABmAjeJyMzAbYwxPzHGzDfGzAf+EdhgjDnuVEznrGITRMXwZHUex9r0akApFVmcvCJYBBwwxhwyxvQATwMrz7D9TVijn4Wfio348hbw843VLL0gi4uKw++iRSmlzpWTiaAAqAyY99jLTiMiScBy4Lkh1t8mIltFZGtDQ8OIB3pGPe1QvZ2DyfNpbO/hro9eENzPV0ophzmZCAZrV2mG2PZaYNNQxULGmF8ZY0qMMSXZ2UHu06fyXfD38p5/BgmxUZQUjQ3u5yullMOGM2bxufIA4wPmC4HqIbZdRRgXCyHRrGsqYk5BEjHR2tBKKRVZnDyqbQGmiMhEEYnDOtivHbiRiIwBrgBedDCWc1exCX/+QrbVeplbmB7qaJRSasQ5lgiMMb3AXcArWIPdP2uM2SUit4vI7QGb3gC8aoxpdyqWc9bTDlXbaMwqobvXz7zx6aGOSCmlRpyTRUMYY9YB6wYsWzNg/lHgUSfjOGeV74Hfy87YOQDMKxwT4oCUUmrkaYH3mRzZBBLNG+2TSU+KZUJGUqgjUkqpEaeJ4EwqNkL+fLbU9DCvMF07mFNKRSRNBEPp6QDPVrzjL+WDulYtFlJKRSxNBEPxbAG/l0PJC/AbtKJYKRWxNBEMpWIjSBTveKcAaNNRpVTE0kQwlIqNkDefrbW9FKQnkp0aH+qIlFLKEZoIBuPthKqtULyEUk8T88Zr/YBSKnJpIhiMZwv4emjNXUzl8U4tFlJKRTRNBIOx6wfelxkAzNNEoJSKYJoIBlOxCXLnsr3OhwjM0aajSqkIpolgIG+XVTRUvJQyTzMXZKeQEu9oTxxKKRVSmggGqtoKvm5M8VJKK5v0/gGlVMTTRDBQxUZAqB5jjUimdxQrpSKdJoKBKjZC3lx21FuzekWglIp0mggC9dUPFC2lzNNEXHQU03PTQh2VUko5ShNBoKpt0NsFxUvZUdnEjPw04mJ0FymlIpse5QLZ9QO+8Zews6qZ+Vo/oJRyAU0EgY5shNzZHGyLoaPHp3cUK6VcQRNBn95ua2jK4svYUdkEaEWxUsodNBH0CagfKPM0kRofw6Ss5FBHpZRSjtNE0KdiEyAw4RJKK5uZUziGqCgdmlIpFfk0EfSpeAvGzaYrdgx7alq0fkAp5RqaCAB6e+z6gSXsqWmh12+Yr2MQKKVcQhMBQPV26O0Eu38h0IpipZR7aCIAq1gIoGgJZZ5mslPjyU1LCG1MSikVJJoIwLqRLGcWJGWww9PEvMJ0RLSiWCnlDpoITtYPLKW508uhhnbtcVQp5SqaCKrfB28HFC+lvKoZ0PoBpZS7aCI4stF6Llpy8o7iuXpFoJRyEU0EFRshZyYkZ1LmaaI4M4n0pLhQR6WUUkHj7kTg88LRd6F4KQCllc1aLKSUch13J4LqHeBth6Il1LV0UdvSpXcUK6Vcx92JIOD+gb4byfSOYqWU27g8EWyE7OmQkk2pp4noKGFWviYCpZS7uDcR+LxQ2V8/UOZpZtq4VBJio0McmFJKBZejiUBElovIPhE5ICL3D7HNMhHZISK7RGSDk/GcoqYUetqgeCnGGEorm7SiWCnlSjFOvbGIRAMPAlcBHmCLiKw1xuwO2CYd+AWw3BhzVERynIrnNAH1AxWNHbR09eodxUopV3LyimARcMAYc8gY0wM8DawcsM3NwPPGmKMAxph6B+M5VcUmyJoGKTna46hSytWcTAQFQGXAvMdeFmgqMFZE1ovINhH54mBvJCK3ichWEdna0NBw/pH5euHoOyfrB3ZUNpEYG82UnJTzf2+llBplnEwEg3XfaQbMxwAXAtcAnwD+WUSmnvYiY35ljCkxxpRkZ2eff2Qn6weWAFDmaWJ2QRox0e6tO1dKuZeTRz4PMD5gvhCoHmSbl40x7caYY8CbwDwHY7Kc7F9oKV6fn13VLczTG8mUUi7lZCLYAkwRkYkiEgesAtYO2OZF4DIRiRGRJOBiYI+DMVkqNkLWVEgdx77aVrp7/czV+gGllEs51mrIGNMrIncBrwDRwCPGmF0icru9fo0xZo+IvAyUAX7g18aYcqdiAqz6gSPvwJwbASj1NAEwX68IlFIu5VgiADDGrAPWDVi2ZsD8T4CfOBnHKWrLoKe1/0ayymbGJsUyPiMxaCEopVQ4cV/taIVdP9DX46inibk6NKVSysXclwiObILMCyA1l46eXj6oa9X7B5RSruauROD3wZG3T14NlFe14DfoHcVKKVdzVyKoLYPuFijqG4imCUDHIFBKuZq7EkHFJuvZvpGs1NNEQXoi2anxIQxKKaVCy2WJYCNkTIa0fMBKBPN0IBqllMu5JxGcrB+wrgYa27qpPN6pdxQrpVzPPYmgrhy6m6H4MgDKqpoBrR9QSin3JIITRyAuBYrs+oHKJkRgjrYYUkq5nKN3FoeVmdfBtKsh2vrKZZ5mLshOISXePbtAKaUG454rAjiZBHRoSqWU6ueuRGDznOiksb1HE4FSSuHSRFDmsSqK9Y5ipZRyaSIo9TQRFx3F9Ny0UIeilFIh585EUNnEjPw04mJc+fWVUuoUrjsS+vyGnVXNzNdiIaWUAlyYCA42tNHR49MbyZRSyua6RLDD7nFUWwwppZTFdYmgtLKJ1PgYJmUlhzoUpZQKC65LBGWeZuYUjiEqSoemVEopcFki6PL62FPTosVCSikVwFWJYE9NC71+ozeSKaVUAFclglKtKFZKqdO4KxF4mslJjSc3LSHUoSilVNhwWSJoYm5hOiJaUayUUn1ckwiaO70camhnvo5RrJRSp3BNIijXoSmVUmpQrkkE8TFRXDk9h7naYkgppU7hmnEaS4oz+M2XMkIdhlJKhR3XXBEopZQanCYCpZRyOU0ESinlcpoIlFLK5TQRKKWUy2kiUEopl9NEoJRSLqeJQCmlXE6MMaGO4UMRkQbgyDm+PAs4NoLhjLRwjw/CP0aN7/xofOcnnOMrMsZkD7Zi1CWC8yEiW40xJaGOYyjhHh+Ef4wa3/nR+M5PuMc3FC0aUkopl9NEoJRSLue2RPCrUAdwFuEeH4R/jBrf+dH4zk+4xzcoV9URKKWUOp3brgiUUkoNoIlAKaVcLiITgYgsF5F9InJARO4fZL2IyAP2+jIRWRjE2MaLyBsiskdEdonINwbZZpmINIvIDvvx7WDFZ39+hYjstD976yDrQ7n/pgXslx0i0iIi9wzYJuj7T0QeEZF6ESkPWJYhIn8Rkf3289ghXnvG36uD8f1ERPbaf8MXRCR9iNee8ffgYHzfFZGqgL/j1UO8NlT775mA2CpEZMcQr3V8/503Y0xEPYBo4CAwCYgDSoGZA7a5GvgzIMBi4N0gxpcHLLSnU4EPBolvGfBSCPdhBZB1hvUh23+D/K1rsW6UCen+Ay4HFgLlAcv+H3C/PX0/8OMhvsMZf68OxvdxIMae/vFg8Q3n9+BgfN8F/u8wfgMh2X8D1v878O1Q7b/zfUTiFcEi4IAx5pAxpgd4Glg5YJuVwGPGshlIF5G8YARnjKkxxmy3p1uBPUBBMD57BIVs/w1wJXDQGHOud5qPGGPMm8DxAYtXAr+1p38LXD/IS4fze3UkPmPMq8aYXnt2M1A40p87XEPsv+EI2f7rIyICfBZ4aqQ/N1giMREUAJUB8x5OP9AOZxvHiUgxsAB4d5DVl4hIqYj8WURmBTcyDPCqiGwTkdsGWR8W+w9YxdD/fKHcf33GGWNqwDoBAHIG2SZc9uWXsa7yBnO234OT7rKLrh4ZomgtHPbfZUCdMWb/EOtDuf+GJRITgQyybGAb2eFs4ygRSQGeA+4xxrQMWL0dq7hjHvDfwB+CGRuwxBizEFgB3Ckilw9YHw77Lw64DvjdIKtDvf8+jHDYl98EeoEnhtjkbL8HpzwETAbmAzVYxS8DhXz/ATdx5quBUO2/YYvEROABxgfMFwLV57CNY0QkFisJPGGMeX7gemNMizGmzZ5eB8SKSFaw4jPGVNvP9cALWJffgUK6/2wrgO3GmLqBK0K9/wLU9RWZ2c/1g2wT6t/ircAngVuMXaA90DB+D44wxtQZY3zGGD/w8BCfG+r9FwN8CnhmqG1Ctf8+jEhMBFuAKSIy0T5rXAWsHbDNWuCLduuXxUBz3yW80+zyxN8Ae4wx/zHENrn2dojIIqy/U2OQ4ksWkdS+aawKxfIBm4Vs/wUY8iwslPtvgLXArfb0rcCLg2wznN+rI0RkOXAfcJ0xpmOIbYbze3AqvsB6pxuG+NyQ7T/bx4C9xhjPYCtDuf8+lFDXVjvxwGrV8gFWa4Jv2stuB263pwV40F6/EygJYmxLsS5dy4Ad9uPqAfHdBezCagGxGbg0iPFNsj+31I4hrPaf/flJWAf2MQHLQrr/sJJSDeDFOkv9CpAJvA7st58z7G3zgXVn+r0GKb4DWOXrfb/DNQPjG+r3EKT4/tf+fZVhHdzzwmn/2csf7fvdBWwb9P13vg/tYkIppVwuEouGlFJKfQiaCJRSyuU0ESillMtpIlBKKZfTRKCUUi6niUCpIBKrZ9SXQh2HUoE0ESillMtpIlBqECLyeRF5z+5D/pciEi0ibSLy7yKyXUReF5Fse9v5IrI5oF//sfbyC0TkNbvzu+0iMtl++xQR+b1YYwE80XcXtFKhoolAqQFEZAbwOazOwuYDPuAWIBmrf6OFwAbgO/ZLHgPuM8bMxboTtm/5E8CDxur87lKsO1PB6nH2HmAm1p2nSxz+SkqdUUyoA1AqDF0JXAhssU/WE7E6jPPT37nY48DzIjIGSDfGbLCX/xb4nd2/TIEx5gUAY0wXgP1+7xm7bxp7VKtiYKPj30qpIWgiUOp0AvzWGPOPpywU+ecB252pf5YzFfd0B0z70P9DFWJaNKTU6V4HbhSRHDg59nAR1v/LjfY2NwMbjTHNwAkRucxe/gVgg7HGmPCIyPX2e8SLSFIwv4RSw6VnIkoNYIzZLSLfwhpVKgqrx8k7gXZglohsA5qx6hHA6mJ6jX2gPwSstpd/AfiliHzPfo/PBPFrKDVs2vuoUsMkIm3GmJRQx6HUSNOiIaWUcjm9IlBKKZfTKwKllHI5TQRKKeVymgiUUsrlNBEopZTLaSJQSimX+/8v0/9+M9QeWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.plot(res[0], label='train')\n",
    "plt.plot(res[2], label='valid')\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend(loc='best')\n",
    "plt.savefig('fig-res-googlenet-train-validate-loss.pdf')\n",
    "plt.show()\n",
    "\n",
    "plt.plot(res[1], label='train')\n",
    "plt.plot(res[3], label='valid')\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('Acc')\n",
    "plt.legend(loc='best')\n",
    "plt.savefig('fig-res-googlenet-train-validate-acc.pdf')\n",
    "plt.show()\n",
    "\n",
    "# save raw data\n",
    "import numpy\n",
    "numpy.save('fig-res-googlenet_data.npy', res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GoogLeNet 加入了更加结构化的 Inception 块使得我们能够使用更大的通道，更多的层，同时也控制了计算量。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 练习\n",
    "GoogLeNet 有很多后续的版本，尝试看看论文，并亲自实现，看看有什么不同\n",
    "* v1：最早的版本  \n",
    "* v2：加入 batch normalization 加快训练  \n",
    "* v3：对 inception 模块做了调整  \n",
    "* v4：基于 ResNet 加入了 残差连接\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考资料\n",
    "* [深入理解GoogLeNet结构](https://zhuanlan.zhihu.com/p/32702031)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}